{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw15_reinforcement_learning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Iallen520/lhy_DL_Hw/blob/master/hw15_reinforcement_learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fp30SB4bxeQb",
        "colab_type": "text"
      },
      "source": [
        "# **Homework 15 - Reinforcement Learning**\n",
        "\n",
        "若有任何問題，歡迎來信至助教信箱 ntu-ml-2020spring-ta@googlegroups.com\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yXsnCWPtWSNk",
        "colab_type": "text"
      },
      "source": [
        "## 前置作業\n",
        "\n",
        "首先我們需要安裝必要的系統套件及 PyPi 套件。\n",
        "gym 這個套件由 OpenAI 所提供，是一套用來開發與比較 Reinforcement Learning 演算法的工具包（toolkit）。\n",
        "而其餘套件則是為了在 Notebook 中繪圖所需要的套件。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5e2bScpnkVbv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!apt update\n",
        "!apt install python-opengl xvfb -y\n",
        "!pip install gym[box2d] pyvirtualdisplay piglet"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_-i3cdoYsks",
        "colab_type": "text"
      },
      "source": [
        "接下來，設置好 virtual display，並引入所有必要的套件。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nl2nREINDLiw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "from pyvirtualdisplay import Display\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()\n",
        "\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "from torch.distributions import Categorical"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "He0XDx6bzjgC",
        "colab_type": "text"
      },
      "source": [
        "最後，引入 OpenAI 的 gym，並建立一個 [Lunar Lander](https://gym.openai.com/envs/LunarLander-v2/) 環境。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N_4-xJcbBt09",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "import gym\n",
        "env = gym.make('LunarLander-v2')"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NrkVvTrvWZ5H",
        "colab_type": "text"
      },
      "source": [
        "## 什麼是 Lunar Lander？\n",
        "\n",
        "“LunarLander-v2” 這個環境是在模擬登月小艇降落在月球表面時的情形。\n",
        "這個任務的目標是讓登月小艇「安全地」降落在兩個黃色旗幟間的平地上。\n",
        "> Landing pad is always at coordinates (0,0).\n",
        "> Coordinates are the first two numbers in state vector.\n",
        "\n",
        "![](https://gym.openai.com/assets/docs/aeloop-138c89d44114492fd02822303e6b4b07213010bb14ca5856d2d49d6b62d88e53.svg)\n",
        "\n",
        "所謂的「環境」其實同時包括了 agent 和 environment。\n",
        "我們利用 `step()` 這個函式讓 agent 行動，而後函式便會回傳 environment 給予的 observation/state（以下這兩個名詞代表同樣的意思）和 reward。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIbp82sljvAt",
        "colab_type": "text"
      },
      "source": [
        "### Observation / State\n",
        "\n",
        "首先，我們可以看看 environment 回傳給 agent 的 observation 究竟是長什麼樣子的資料："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rsXZra3N9R5T",
        "colab_type": "code",
        "outputId": "f0604c7f-48d6-49cb-8921-6407b3f6c4f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "tags": []
      },
      "source": [
        "print(env.observation_space)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Box(8,)\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezdfoThbAQ49",
        "colab_type": "text"
      },
      "source": [
        "`Box(8,)` 說明我們會拿到 8 維的向量作為 observation，其中包含：垂直及水平座標、速度、角度、加速度等等，這部分我們就不細說。\n",
        "\n",
        "### Action\n",
        "\n",
        "而在 agent 得到 observation 和 reward 以後，能夠採取的動作有："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p1k4dIrBAaKi",
        "colab_type": "code",
        "outputId": "11c1a4d8-6455-451f-9999-16a5eec8fa63",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "tags": []
      },
      "source": [
        "print(env.action_space)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Discrete(4)\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dejXT6PHBrPn",
        "colab_type": "text"
      },
      "source": [
        "`Discrete(4)` 說明 agent 可以採取四種離散的行動：\n",
        "- 0 代表不採取任何行動\n",
        "- 2 代表主引擎向下噴射\n",
        "- 1, 3 則是向左右噴射\n",
        "\n",
        "接下來，我們嘗試讓 agent 與 environment 互動。\n",
        "在進行任何操作前，建議先呼叫 `reset()` 函式讓整個「環境」重置。\n",
        "而這個函式同時會回傳「環境」最初始的狀態。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pi4OmrmZgnWA",
        "colab_type": "code",
        "outputId": "b1fbd5cf-dfcd-40a1-f5e8-4ea078aec347",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "tags": []
      },
      "source": [
        "initial_state = env.reset()\n",
        "print(initial_state)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "[-0.00720987  1.3988848  -0.73030055 -0.5349194   0.00836124  0.16542402\n  0.          0.        ]\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uBx0mEqqgxJ9",
        "colab_type": "text"
      },
      "source": [
        "接著，我們試著從 agent 的四種行動空間中，隨機採取一個行動"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vxkOEXRKgizt",
        "colab_type": "code",
        "outputId": "e4921d68-698f-46fe-f608-27fd18c4c78c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "tags": []
      },
      "source": [
        "random_action = env.action_space.sample()\n",
        "print(random_action)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "1\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mns-bO01g0-J",
        "colab_type": "text"
      },
      "source": [
        "再利用 `step()` 函式讓 agent 根據我們隨機抽樣出來的 `random_action` 動作。\n",
        "而這個函式會回傳四項資訊：\n",
        "- observation / state\n",
        "- reward\n",
        "- 完成與否\n",
        "- 其餘資訊"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_WViSxGgIk9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "observation, reward, done, info = env.step(random_action)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FdieGq7NuBIm",
        "colab_type": "text"
      },
      "source": [
        "第一項資訊 `observation` 即為 agent 採取行動之後，agent 對於環境的 observation 或者說環境的 state 為何。\n",
        "而第三項資訊 `done` 則是 `True` 或 `False` 的布林值，當登月小艇成功著陸或是不幸墜毀時，代表這個回合（episode）也就跟著結束了，此時 `step()` 函式便會回傳 `done = True`，而在那之前，`done` 則保持 `False`。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yK7r126kuCNp",
        "colab_type": "code",
        "outputId": "3b232447-c2a2-498e-a0ae-fe16c3396a05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "tags": []
      },
      "source": [
        "print(done)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "False\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKdS8vOihxhc",
        "colab_type": "text"
      },
      "source": [
        "### Reward\n",
        "\n",
        "而「環境」給予的 reward 大致是這樣計算：\n",
        "- 小艇墜毀得到 -100 分\n",
        "- 小艇在黃旗幟之間成功著地則得 100~140 分\n",
        "- 噴射主引擎（向下噴火）每次 -0.3 分\n",
        "- 小艇最終完全靜止則再得 100 分\n",
        "\n",
        "> Reward for moving from the top of the screen to landing pad and zero speed is about 100..140 points.\n",
        "> If lander moves away from landing pad it loses reward back.\n",
        "> Episode finishes if the lander crashes or comes to rest, receiving additional -100 or +100 points.\n",
        "> Each leg ground contact is +10.\n",
        "> Firing main engine is -0.3 points each frame.\n",
        "> Solved is 200 points."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vxQNs77hi0_7",
        "colab_type": "code",
        "outputId": "44c84a9f-499a-4e3d-d1d8-e565791a5f89",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "tags": []
      },
      "source": [
        "print(reward)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "-1.821301334364988\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mhqp6D-XgHpe",
        "colab_type": "text"
      },
      "source": [
        "### Random Agent\n",
        "\n",
        "最後，在進入實做之前，我們就來看看這樣一個 random agent 能否成功登陸月球："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y3G0bxoccelv",
        "colab_type": "code",
        "outputId": "67f637a7-9c90-404c-d53c-6c1d54ba30a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "env.reset()\n",
        "\n",
        "img = plt.imshow(env.render(mode='rgb_array'))\n",
        "\n",
        "done = False\n",
        "while not done:\n",
        "    action = env.action_space.sample()\n",
        "    observation, reward, done, _ = env.step(action)\n",
        "    img.set_data(env.render(mode='rgb_array'))\n",
        "    display.display(plt.gcf())\n",
        "    display.clear_output(wait=True)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5paWqo7tWL2",
        "colab_type": "text"
      },
      "source": [
        "## Policy Gradient\n",
        "\n",
        "現在來搭建一個簡單的 policy network。\n",
        "我們預設模型的輸入是 8-dim 的 observation，輸出則是離散的四個動作之一："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J8tdmeD-tZew",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PolicyGradientNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.fc1 = nn.Linear(8, 16)\n",
        "        self.fc2 = nn.Linear(16, 16)\n",
        "        self.fc3 = nn.Linear(16, 4)\n",
        "\n",
        "    def forward(self, state):\n",
        "        hid = torch.tanh(self.fc1(state))\n",
        "        hid = torch.tanh(self.fc2(hid))\n",
        "        return F.softmax(self.fc3(hid), dim=-1)"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ynbqJrhIFTC3",
        "colab_type": "text"
      },
      "source": [
        "再來，搭建一個簡單的 agent，並搭配上方的 policy network 來採取行動。\n",
        "這個 agent 能做到以下幾件事：\n",
        "- `learn()`：從記下來的 log probabilities 及 rewards 來更新 policy network。\n",
        "- `sample()`：從 environment 得到 observation 之後，利用 policy network 得出應該採取的行動。\n",
        "而此函式除了回傳抽樣出來的 action，也會回傳此次抽樣的 log probabilities。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zZo-IxJx286z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PolicyGradientAgent():\n",
        "    def __init__(self, network):\n",
        "        self.network = network\n",
        "        self.optimizer = optim.SGD(self.network.parameters(), lr=0.001)\n",
        "\n",
        "    def learn(self, log_probs, rewards):\n",
        "        loss = (-log_probs * rewards).sum()\n",
        "        self.optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        self.optimizer.step()\n",
        "\n",
        "    def sample(self, state):\n",
        "        action_prob = self.network(torch.FloatTensor(state))\n",
        "        action_dist = Categorical(action_prob)\n",
        "        action = action_dist.sample()\n",
        "        log_prob = action_dist.log_prob(action)\n",
        "        return action.item(), log_prob"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehPlnTKyRZf9",
        "colab_type": "text"
      },
      "source": [
        "最後，建立一個 network 和 agent，就可以開始進行訓練了。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GfJIvML-RYjL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "network = PolicyGradientNetwork()\n",
        "agent = PolicyGradientAgent(network)"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouv23glgf5Qt",
        "colab_type": "text"
      },
      "source": [
        "## 訓練 Agent\n",
        "\n",
        "現在我們開始訓練 agent。\n",
        "透過讓 agent 和 environment 互動，我們記住每一組對應的 log probabilities 及 reward，並在成功登陸或者不幸墜毀後，回放這些「記憶」來訓練 policy network。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vg5rxBBaf38_",
        "colab_type": "code",
        "outputId": "a2ac2226-a803-41c2-dcc2-9b338ad591d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "tags": []
      },
      "source": [
        "agent.network.train()  # 訓練前，先確保 network 處在 training 模式\n",
        "EPISODE_PER_BATCH = 5  # 每蒐集 5 個 episodes 更新一次 agent\n",
        "NUM_BATCH = 400        # 總共更新 400 次\n",
        "\n",
        "avg_total_rewards, avg_final_rewards = [], []\n",
        "\n",
        "for batch in range(NUM_BATCH):\n",
        "    log_probs, rewards = [], []\n",
        "    total_rewards, final_rewards = [], []\n",
        "\n",
        "    # 蒐集訓練資料\n",
        "    for episode in range(EPISODE_PER_BATCH):\n",
        "        state = env.reset()\n",
        "        total_reward, total_step = 0, 0\n",
        "\n",
        "        while True:\n",
        "            action, log_prob = agent.sample(state)\n",
        "            next_state, reward, done, _ = env.step(action)\n",
        "\n",
        "            log_probs.append(log_prob)\n",
        "            state = next_state\n",
        "            total_reward += reward\n",
        "            total_step += 1\n",
        "\n",
        "            if done:\n",
        "                final_rewards.append(reward)\n",
        "                total_rewards.append(total_reward)\n",
        "                rewards.append(np.full(total_step, total_reward))  # 設定同一個 episode 每個 action 的 reward 都是 total reward\n",
        "                break\n",
        "\n",
        "    # 紀錄訓練過程\n",
        "    avg_total_reward = sum(total_rewards) / len(total_rewards)\n",
        "    avg_final_reward = sum(final_rewards) / len(final_rewards)\n",
        "    avg_total_rewards.append(avg_total_reward)\n",
        "    avg_final_rewards.append(avg_final_reward)\n",
        "    print(\"Batch {},\\tTotal Reward = {:.1f},\\tFinal Reward = {:.1f}\".format(batch+1, avg_total_reward, avg_final_reward))\n",
        "\n",
        "    # 更新網路\n",
        "    rewards = np.concatenate(rewards, axis=0)\n",
        "    rewards = (rewards - np.mean(rewards)) / (np.std(rewards) + 1e-9)  # 將 reward 正規標準化\n",
        "    agent.learn(torch.stack(log_probs), torch.from_numpy(rewards))"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Batch 1,\tTotal Reward = -247.8,\tFinal Reward = -100.0\nBatch 2,\tTotal Reward = -168.8,\tFinal Reward = -100.0\nBatch 3,\tTotal Reward = -113.5,\tFinal Reward = -100.0\nBatch 4,\tTotal Reward = -157.5,\tFinal Reward = -100.0\nBatch 5,\tTotal Reward = -150.4,\tFinal Reward = -100.0\nBatch 6,\tTotal Reward = -121.9,\tFinal Reward = -100.0\nBatch 7,\tTotal Reward = -131.5,\tFinal Reward = -100.0\nBatch 8,\tTotal Reward = -158.6,\tFinal Reward = -100.0\nBatch 9,\tTotal Reward = -141.4,\tFinal Reward = -100.0\nBatch 10,\tTotal Reward = -115.5,\tFinal Reward = -100.0\nBatch 11,\tTotal Reward = -190.8,\tFinal Reward = -100.0\n"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-36-8e16d009ed62>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     39\u001b[0m     \u001b[0mrewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m     \u001b[0mrewards\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1e-9\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# 將 reward 正規標準化\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 41\u001b[0;31m     \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlog_probs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_numpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-34-61eb1c64e96b>\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, log_probs, rewards)\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mlog_probs\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m         \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/.local/lib/python3.6/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    148\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m         \"\"\"\n\u001b[0;32m--> 150\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    151\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m~/.local/lib/python3.6/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[1;32m     98\u001b[0m         \u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m         allow_unreachable=True)  # allow_unreachable flag\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNb_tuFYhKVK",
        "colab_type": "text"
      },
      "source": [
        "### 訓練結果\n",
        "\n",
        "訓練過程中，我們持續記下了 `avg_total_reward`，這個數值代表的是：每次更新 policy network 前，我們讓 agent 玩數個回合（episodes），而這些回合的平均 total rewards 為何。\n",
        "理論上，若是 agent 一直在進步，則所得到的 `avg_total_reward` 也會持續上升，直至 250 上下。\n",
        "若將其畫出來則結果如下："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZYOI8H10SHN",
        "colab_type": "code",
        "outputId": "1f1cae34-df0c-4bd5-9d89-c5ed57f4a603",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "plt.plot(avg_total_rewards)\n",
        "plt.title(\"Total Rewards\")\n",
        "plt.show()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd5gkVbn/v29Vp8mzYTbvsssm8hJWsoDkpCj3iqBXDFf5qaAXM4iBe68gIsoVUS+I+YoICILkICxB0pJZYJdNbN6d3dnJMx2qzu+Pqrfq1OmqDjPTMz0z5/M8++x0dXXX6equ73nre97zHhJCQKPRaDTjC2OkG6DRaDSa4UeLv0aj0YxDtPhrNBrNOESLv0aj0YxDtPhrNBrNOESLv0aj0YxDtPhrxj1EJIhowUi3Y6AQ0XFEtGmk26EZXWjx11QtRNQt/bOJqE96/LGI1wypEBLR40TU7x5zJxHdQUTTh+r9NZqRQou/pmoRQtTzPwAbALxf2vanYWzKRW4bFgCoB3DNMB47ABHFRurYmrGFFn/NqIOIkkT0P0S0xf33P+62OgD3A5gh3SHMIKJDiegZImonoq1EdD0RJco9rhCiHcDfABwotWUvInqYiNqIaCURneNun+cez3Af/4qIdkiv+yMRXez+/SkieouIuohoLRH9P2m/44hoExF9k4i2AfgtEdUQ0e+IaDcRvQngPcr5+SYRbXbfbyURnVDuZ9WMfbT4a0YjlwE4HI4ILwFwKIBvCyF6AJwGYIt0h7AFgAXgywAmAzgCwAkAvlDuQYloEoCzAax2H9cBeBjAzQCmADgXwC+IaB8hxDoAnQAOcl9+DIBuItrbfXwsgGXu3zsAnAmgEcCnAFxLRAdLh54GYCKAPQBcAOB7AOa7/04B8AmpjYsBXATgPUKIBvf59eV+Vs3YR4u/ZjTyMQD/JYTYIYRoBfCfAD4etbMQ4kUhxLNCiJwQYj2AG+CIb6lcR0QdAHbC6UC+6G4/E8B6IcRv3fd+GcBfAXzYfX4ZgGOJaJr7+Hb38Tw4Qv+q2757hRBrhMMyAA8BeK90fBvA94QQaSFEH4BzAFwhhGgTQmwEcJ20rwUgCWAfIooLIdYLIdaU8Vk14wQt/prRyAwA70qP33W3hUJEi4joHiLaRkSdAK6EI+Kl8iUhRBOAAwBMADDL3b4HgMNce6ediNrhdEws9ssAHAcn6n8CwONwOp1jATwphLDd9p1GRM+61lE7gNOV9rUKIfqVz79R+fwAACHEagAXA7gcwA4iuoWIIs+NZvyixV8zGtkCR3iZOe42AAgrU/tLAG8DWCiEaATwLQBU7kGFEK8D+D6AnxMRwRHgZUKIZulfvRDi8+5LlsGJ4I9z/34KwFGQLB8iSsK5W7gGwFQhRDOA+5T2qZ9pK4DZ0uM5SjtvFkIcDeccCQA/LPezasY+Wvw1o5E/A/g2EbUQ0WQA3wXwf+5z2wFMIqImaf8GOP57NxHtBeDzGDi/BzAVwAcA3ANgERF9nIji7r/3sK8vhHgHQB+Af4PTSXS67fsX+H5/Ao5N0wogR0SnATi5SBtuBXApEU0golnwbSgQ0WIiOt7tVPrd49uD+LyaMYoWf81o5PsAlgN4DcDrAF5yt0EI8TaczmGta8XMAPA1AB8F0AXgVwD+MtADCyEyAH4K4DtCiC44Qn0unDuPbXCi7KT0kmUAdrnePD8mt81w3+NLcAR9t9vOu4s04z/hWD3r4IwP/FF6LgngKjjjE9vgDERfOoCPqhnjkF7MRaPRaMYfOvLXaDSacYgWf41GoxmHaPHXaDSacYgWf41GoxmHjJoiUZMnTxZz584d6WZoNBrNqOHFF1/cKYRoCXtu1Ij/3LlzsXz58pFuhkaj0YwaiOjdqOe07aPRaDTjEC3+Go1GMw7R4q/RaDTjkCERfyL6DRHtIKI3pG0T3UUu3nH/n+BuJyK6johWE9FrSt1yjUaj0QwDQxX5/w7Aqcq2SwA8KoRYCOBR9zHgLLax0P13AZyKixqNRqMZRoZE/IUQTwBoUzafBacCItz/Pyht/4O7cMWzAJr1gtgajUYzvFTS858qhNjq/r0NThlcAJiJ4EIUm9xteRDRBUS0nIiWt7a2Vq6lGo1GM84YlgFf4ZQOLbt8qBDiRiHEUiHE0paW0HkKGo1GM+K8sbkDL23YPdLNKItKiv92tnPc/3e42zcjuArRLHebRqPRjErO/NlTOPsX/xzpZpRFJcX/bgCfcP/+BIC7pO3nu1k/hwPokOwhjUaj0QwDQ1LegYj+DGed0slEtAnA9+CsJnQrEf07nFWHznF3vw/OAtWrAfQC+NRQtEGj0Wg0pTMk4i+EOC/iqRNC9hUALhyK42o0Go1mYOgZvhqNRjMO0eKv0Wg04xAt/hqNRjMO0eKv0Wg04xAt/hqNRjMO0eKv0Wg0g8C2yy5eUBVo8ddoNJpBkLHskW7CgNDir9FoNIMgndXir9FoNOOOdM4q+Pyiy+7H+b95fphaUzpa/DUajWYQpHOFI/+MZeOJVdVXkl6Lv0aj0QyCYpF/taLFX6PRaAZBv/b8NRqNZmBcce+b+MnDq0a6GQNCZ/toNBrNAPnVk+tw3aPvjHQzBoTO9tFoNJpxiPb8NRqNZhxSLNunWtHir9FoNINAFn9nrarRgRZ/jUajGQTprG/7FCrzs6a1exhaUzpa/DUajWYQyJG/VUD9T/zJMmxs6x2OJpWEFn+NRqOJ4IlVrfjDM+sL7iOLv13A9hECaOvJYGNbb1UMEmvx12g0I0o1++Tn/+Z5fPeuFQX3kYW8UOQPAO19Wbz36sdwyV9fH5L2DQYt/hqNZkTJjdJ6+Ex/Rvb8C3+Wjr4sAOCxlTsq2qZS0OKv0WiGhWseXImP3fRs3vZMFaZK/uWFDXh9U0dJ+/bK4l/ko1TTZ42NdAM0Gs344PrHVoduryZBZL5Zhi3TJ2X7WErkr67yJe870lRc/IloPYAuABaAnBBiKRFNBPAXAHMBrAdwjhBid6XbotFoqg+5No4QAkQ0gq0Jp1C7+jLRnr9qA/VlcgCAaviEw2X7vE8IcaAQYqn7+BIAjwohFgJ41H2s0WjGIXLkn7Wq0/8vNItXtn3UwWv1TkDed6QZKc//LAC/d//+PYAPjlA7NBrNCJMOiP/IW0C5kDYUqtzZW9D2UfZ1xb8a7m6GQ/wFgIeI6EUiusDdNlUIsdX9exuAqWEvJKILiGg5ES1vba2+lXA0Gs3gyVSZ+IcJvVq588l3WnHpHc64QH8B2yc/8s8NVTMHzXCI/9FCiIMBnAbgQiI6Rn5SOPdJofd6QogbhRBLhRBLW1pahqGpGo1muJHz5DOWjVXbu7BiS2mZNhVpT0iJZnVS1lOrd+IvL2wAAPRmfUFXI33V8+9Nu5H/UDR0kFRc/IUQm93/dwC4E8ChALYT0XQAcP8f+aRXjUYzIqie/8nXPoEzrntq5NoTFvkrnr8QTh2fnGUHUz2LZPuMG8+fiOqIqIH/BnAygDcA3A3gE+5unwBwVyXbodFoqhdZbLNVkPYZlnqqbmNRz1g2+jMWEjFHSlWbR7WBehTb57m1u3DPa1vwbzc9V3R28FBT6VTPqQDudAc3YgBuFkI8QEQvALiViP4dwLsAzqlwOzQaTZVg2wKG4Rsf1eb5h2X2qNtYpzM5G71ZCw3JGHblMnmRvqrnfd6Ar3MePnKjP+ltV3caUxpTQ/AJSqOi4i+EWAtgScj2XQBOqOSxNRpNdZKzBRIR4l8N6+GGRf5pZXKWcIcp0znH9pnelMKungwsIfDkO63YtLsP5x06J88G6pFsn35lHGG4F4XRM3w1mnHK1o4+1CZiaKqJD+txVXsjYPtUQZ5/WAekbmNN78tYyORs1CUcKbVt4N9/txwZy8YRe07y7CDGz/Yh9KQt5bnhHQ/QtX00mnHKp3+3HD968O1hP25WSYmJyvMfbg+cCY/8VdvHaRsXaqtPxbztC6fWAwD+9Ny7+dk+ksCraZ/d6eFNA9Xir9GMU3Z1p9HWkxmWY8kzXy0lug94/tLfUXVwNrb14qKbX0J/herkhNXaz/f8FfFPOuJv2QKNKedOaldPJn+SlyTwaqQ/3HMAtPhrNOOUdM4OzWkfCnZ2p3Hmz570Vq6SyzarJZzTEZ5/lBh+9dZXcc9rW/HShsqUAwvN9rGCQs0foV0VfyE8L78vY+VP8spGR/49OvLXaDTDQTpnVWyA9c6XNuONzZ347dPrAQQFVbZz+rMW/vjMeu+x7Pn3RXjgvBZu3KyMfJVi+/CdTEevc+dU54q/EMLbty9r5VlXcl+gev7dae35azSaMhFC4OoH3sa6nT0l75+pYOTPvn485mT1RKVz3vHSZqzf1Rv6XNQA6C7XqorqHAZLKZO82M5h26chxbaPbxv1ZazIVcqIdOSv0WiGgF09Gfzi8TV44I1tJe2fswVsEe5vDwU5N4KPG47EZCIGcjPK8YuJvyyYlcqOCUu5/N7dK3DXK5u9x+z5d/U77UnFTQDOZ+PXP7euDSdd+0TkcfIjfy3+Gs2I8Pa2ztCKjqMBFpyu/mzB/XKWjf++501s2t0XeN1QwyIeM/Mjf9nz5yh+mju5Sd4vzPPf0Zn2/u7LVkYsoxaX+d0/13t/80fgGbtJN6XTFqKkc0oI+v8A8KMHV+Le17aGv6ACaPHXaAC0dqVx+k+fxMNvbh/ppgwIznzpLCL+63f14NdPrcN9rzsiowpdzrKLdiClwN49+/LpCM+/tSuNloYk7r7oKKc9RSJ/HmAFgL5MZTquKPHmuxjA9/y5jQHxLzELqTck0r/w5pfKautg0OKv0cCJmG0RFJfRBHv3bENEwVH31o7wyP8rt76K/S9/CF39WXzl1ldw+d0rIn3rgsfhyN8o7Pnv7E6jpT7pdRKyjx/m6bf3+qmplUqNjIr8TWlWMts+XKXTq+1jlxj5U3C2L1Pj2kfDgRZ/jQZ+NKqmIY4WOL2ws0jnxV781vZ+APnif/erWwAAr2xsxx0vbcbv/rl+QNYQCzyfT3lsQY38JzckEXfFU/bBwyL/jkDkX6EB34jPKwcG/BHYukmY5Yk/EB7592UtzL3k3sDnrBRa/DUa+DbFqPX8S4z8WZS3dLD4hwuoPPlrIMXWMu75ZCGVBXXFlk4c/+PHsaOr37F96pOIu2MDwQHd/M/S3uuLouqZDxVqTj+zvbPf+9uP/F3PP+5Iab/7PZSyUFdY5M9s6+iPfG6o0OKv0QDIubl7I1VSYLB4kX8Rv94qYvswsvgP5JywcPP7y17+LS9swNrWHvzfsxuwszuDloYkEqYBIqBT6rzCbR/n89UmzGGP/Nt6Mt7YivAGfDnyd+wa/tzNReolEchbzD0MHiivJFr8NRr49kQ1FBYbCKVH/s7nYxHN5OyAp88ReDDyL/+csH0TFvnPnVQHAHj4ze3IWDYm1ydARKiNm+jo84+rRsZvbO7An5/fgIZUDA2pWJ74CyGwbFVrXlnlclHF/5oPL8G3z9gbgJ9t5EX+roCz588lKZprE0WPUyjyH46VvnRVT40GvhduqcVYRgnpUj3/kM+XsWxc/4/VmNaUQswwkLUsLwUz6jVRrGntxtOrd3oTlthCkQWV2/rW1k4AfmmEumQMu3v89qsifObPnNW9ZjSlkIybebbPnS9vxldufRVXnb0/ahIm3tnejc8es2dJVUs5++n0/afnTfKaN7kOW9o5a8k5pq1k+7D48+Nix1QneX1gyQx09GWxbJWzVvlwjD1p8ddo4Hv9oz3y78lYyFk2YhGlD8JEJZ2z8bN/rAbgzlTNAm3dkviXcU4+eP3T6ErnsN/MRgCOgK/Y0oGtkofdr8wqrnXFvz4Zw+7e4p3Ojq40Fk5tyIv8V27rAgC09WZw9R0rAQD7zmjEaftPL9jmjW29+MKfnBTL9VedkTfrOWEaXqYP1+rxBnzTwTz/NzY7aw9PqC3e4cgdXdw0vPcAyjvnA0WLv0YDXxRHu+cPODNFo2yHMFGRq2NyymVbIPIv/Zx0uWLIwpbO2Tj/188Homm1GmddwvHL65IxtHb5k7iiOuKcLRzPX5nkxVZWc43/2UupXXT7i5uk98ggrbwmHiMY7ggu/z68PP9sMPK/x52kVYrtI5/jRIy8WcJAeXdbA0V7/hoN/ItNrTU/WpCj1c6+aN8/zNaSM0s4L39Xjy/CA8mAYmHr7s9hV08mMBahin9tgm0fE+198h2Hf1x54lkqbqA2YealgvJraxO+iJZyJyd//tc2deTZTXEp8ufTx/0hD5cklDutMNtn7qTawGNZ/OOmgVTcf4/huAPV4q/RQPL8R6ntIwtqoYyfMFHZ0OYXVisn8n/x3d1Y09qNtW6VTRke+NwakrKo2j51SUes65OxwHPycbkcxWeOnoeHLj4WqXh+tg9H/nKnUEqa6vaufuzhCvPrmzvyOifH9nH+9m2f4DlJKit2NSrif8sFh+PLJy3yHnf2ZZGxbG9Sl2P7SJH/MKQca9tHo4EvNKNpkldHXxaf/f1yfOuMvQMpm4XEP8xOkMWfUwx3S/n0Uf7zv/zyn97ff7ngcBy256S8fTilVEZdu5Yjf/6fkYWbxf/MJTMwZ1Kta/sE34cnRsnVMUsR0R2daSxoqcfOLmdxG/WOIhEz8mwf9WeiLteYUFI165MxtEv7cKbPtKYU1u3sQdw0IE0gHpbfoY78NRrIM1IHH3H98IG38fPHVg/6fWTaejJ5dYduf3ETnl/fhp88vCoQrXb159DRm8XXb3sVO7qCkXeYkG+QSirHjPwkw1KssBfWt4V2OnInwlZInuef9D1/ta13vLQJ/7tsDZavb0PMIOzZ4qSJhtk+LP5dkviXYp/s6OrHlMYkahIm+rMWejNWwDqaXJ/0xJ+9frXkhSr+6oC7aVAgsmemNiad15tBz783Y+Ezv38BVz/wdsXGoXTkr9FAKu8wBLbPLx9fAwA4aZ+pWDS1YdDvBwAX/+UVPLGqFc9/6wRMcStg3u2WGN7S3oc9Jvp+8rqdPejozeK2FzfhrW2duOeL7/WeC4so5cg/rIxPmPioEfW6nb044PKHCn6GloYkOvqyeXYNR/z1yaA4ZiwbX7n1VQDAzOYaHLOoxVsiMRkz8zoRtn0CkX+BjsuyBU776RPuRLMUkjHnbqIvk0NjKu51LqZBfraPnW/7GBSs+wMAh86bGHhsGpTXQQB+NdO4O8mN2dDWi0fe2oFH3tqBsw+ehQVT6iM/x0DRkb9GA1/0h/J2+/GVO0rab0dnP/7jlpfxt5c3R+7DWTBbJA99o2uFrN7Rjc3tfZ5/fNX9b+Mbf30NAPDG5k4c/cN/4ObnNgAI/3yrtvuevWwf8QIlYb65Wnvm5Y3Fl1TkAc1+ZUBVzvZhEjEDz6zZ5T3e3N6HU/ad6j02iPI6KraBekqM/Nt7M95nn+pG/umsjd6MlTdg69k+XuTvPxczDJhKPYcpDUm8cNmJ0uvzxwUAYGqTK/4xIxD5y5+hUnX+tfhrxg0dvVl09Ib74WxtDMVAG2d+lLpK1vWPrcZdr2zBna7492etvKi2pcGxBzZKUXp3OudlkGxo68XkhvD0wk27+/CtO18H4H++xpQvtDu7/cweudYPR9lhkf9u5TyubS2+ghifFzWbhi2Sekn8wzz9ORPrvL9NIxh9yzZMd0D8o78D+TMkYyZSccOJ/LMWGmuctsxwxTk/28c/XsykvMifiLzZ0oDTeYTZPnLknwyMCUjiX2TW9kAZMfEnolOJaCURrSaiS0aqHZrRhWUL3PnypgH5oF+97VV8/fZXI98XGHzkn7NsL7e81OqObFew973Xdx7AcT96HNf/4x0c/+PHIYRAS70j/mzRZC0bmZyNaa447exOoy4R86JoFfaw+Q6HrSNm0VTHVpAtGc5YCbPC5NLKUaiRrrzmbljhMznyrw0pbcwdIOCIqfwbkP8ODvhGf5/yhLJjFk1GjZtB1JexvAHd6c01ABCS7eO/j2kQDEn8P7BkBqY1pgIdQpjtEzfJu8NIxAwkx0PkT0QmgJ8DOA3APgDOI6J9RqItmtHF7/65Hl/+y6u4/cWNZb+2tasfrVKUK+NX9Ryc+Mv1WrgTsG2Be17bEtlhcZQv29PbOvtxzUOrsLa1B7t6Mp5YcuTPdeQ5cmzvzSIZNz17QoULjXHnxgONzHvmTsxrf1NNtO2jRv5hcOfR7M52/ehhc7znJoRMgpLFvyakE5sitdkwKBB9W0IW/9JSPTmd9Z4vHo0pDSmk4ia60lnkbIGle0zAnIm1Xk0fPq+2MskLgJup45/36847CKZBgc7OifyDclsTN7191AFf+TNUam3fkYr8DwWwWgixVgiRAXALgLNGqC2aUcTqHc4U/qjKi4VIF1iwnCc/DTbyly9UXtHpthc34qKbX8bNz70b+hr2wK2IRVO2tPd5Fsiq7V0QQqDbtQWmNdV4+yVjRuSM1iZXbNn2WbrHRC/aB4BjF7XkvWZinfOacNuneOTPUe2R8ydh/VVn4NT9pnnP1Sfzc03kAV817RMAGqTXGBSMvuU2dpfo+e92xX+C+zlTcdObmdzSkMQT33gfDpozAQBCBnz99zENyvP8gWDmlBES+ack8Y+bBuLS/t1jNfIHMBOAHLptcrdpNAVpUy7Ycsjk7Ehx9CL/QaZ6yhcq2z5b3IVTWrvDBTPtlQmOFn/e56UN7fi/5zZ4ncz0Jt++ScXNwOeTBZbtIO7cLj5xIR74j2NwwTF74q4Lj8K8yb6fziyY4mQqZUPEvxTbhyN5fu+YtAxiPKRk8aQ6P7IPi/xJElizkO0j+eW/eXodLr3j9dD28d3LRLdjrImb3u9L7Xy8yD9kklfcIBghShqwfUIi/2Tc9/njpgH5LI9Z26dUiOgCIlpORMtbW1tHujmaKoAvzgGsLOhE/hGLl/hVPQu/8bqdPZh7yb14bVN76POq+H/+/17ETx99BwDyosMbn1iDp1fv9CP/iGNv2u1E/ofsMQF7TKrFP6WqmVMl7z4ZM4JZKJLA8sSqnG3DNAhEjk/9rdP3xpLZzXk59gCw1zRH/NVB8OfXteHK+94ObatMl5sRxCWc5THReEjhuf1mNnl/8xiFOpDKkJJ3H4j8lQHSPz+/IfQ9dvdmkIwZXkfDA77y8RlvwDfM8zfDI3+5szIobCKYEYj87YB1NXbFfzOA2dLjWe62AEKIG4UQS4UQS1ta8m9LNeMPFv9K2T7FygE88MY2AH4BL5mXNuz2rARu4/3u/oA/aMhced/b+NhNz3lRfZhDYRqELe396MtYqImbWDilHut29nie8MS6hBc9cnYOY9sCJ+3jpEfyGEHOEqETucLEn20h1Qp7c0tH3r6PfvVYfPPUvQLbuCz0HHcOgpwBE7ZYiSz0nLY6MeIOT7VhomyfQrT1ZALvL6+fq955+DN8nccBz98wIsdavNeHTPJKxEzvfCRiwdfLn2Gsef4vAFhIRPOIKAHgXAB3j1BbNKMIvlUvpVqjSjpnRWbgZENEBHA6g7BMkjrFFli9oxtn/+Kf+Pbf3gDgiJN6l2GGeQPw7SEhRN7x95hUi988vQ4vbWhHKm5i3uQ6rNvZ4xU6q02YnnDPaA5m8AgAvzp/Kc59z2zPCsnZ4eIfloNenwzP9qEQoZvRVIPPHbsnbrngcG8b2z1zJ8spmq74Gwbu+eLRePxrxwXe57GvHYfL37+PJ77JmIG4STjv0NmB/fxI3Hksj5fw+Qz7TDK7ezKBgWd5wFVdSL1QbR812ycM0+345FOXjBmYO7kOsyfWYM/J9QHbizv3mrg5tiJ/IUQOwEUAHgTwFoBbhRArRqItmpFnY1sv3tneVdK+5Ub+6ZyzIPYNy9YUsX3CB3z3/e6DOPnaZd5jvhDrlNmonIXDhcwm1iXyOpqIEvt+5G+LQH6/QcDJ+/iDpDUJE/Mm1yOds7F6hzM5qT7pr2g1vakG//tvh/gRtPtRahMxKfIPr/WvVqU8ce+pXnQuL3GZydneuT9yvl/LxzCcTuHQuf7M1p+eeyBu/uxhAWuKff+4SdhvZlOgYwCcDuOTR83z2pOIGXjnitPxg7MPCOzHItrZn8UV977pfT6ZsEFlmd29GUyo8++WZPFXbR8120ceGoqypgKvd602+TwnYgamNqbw5DeOx9zJdThi/iT8+hNLAfjjFhPrEmMvz18IcZ8QYpEQYr4Q4oqRasdYQggxoMW2R5r3Xv0YTrr2iaL7yVExC9Dm9j7c+MSayMFSLiV8/T9WI5OzkVaWLWRY9F/e0I6bnlzrH8eysUaawMSRfzJu4lO/fd6rBa/WtZlUl8izmKIyidjzt0VQ/E2DcMlpe3kTjWrihhdNL3/XmVFbl4x5PvX05hRO3W8afvvJ9wDwtB91SRM9mRyEEMjZInSw1TB8S+acpbNw0yeWencIHPl/+H//iUXfvt/rQGWBZFGXI+ApDSkcOX9y4Dhy5F8I7njUTsl7H1eMf/XEWvzqyXX47dPr8vapTYbPeWDSOTvS6lFtn0LlHcLGL1T4tMh3I2F3JsfvNQWA/zubVJ8IDGAPJVU94Kspjx8/tAoLL7s/b3boWEGO2tn2+czvl+PK+97G5vb86pGAP2mJUyOFCE//k62N79/7Fjr7s6GdBEf+j7y5HY+tbMW3/+ZkkqjlDibVJ/IWBVFLGfvb/Tx/ufQB2yu8MEgqbuKAWU2Y3pTylvuTo9sZbton20AsULWJGGzhHD9nichIlYWW7Sm+Q+DI/6UNziA3d7zypKSwtwwTt0Kevwx3DmH1cJzjkdtW5/+w+RuqNaeStexAJ5SSjhWV7RNW3qGUyJ/3SUi+f9j54XERWzjntKkmju6Qu5qhQIv/GIKzGoot4j1aCa4D6/zNa9ZGZf9wcS75+TDrR43KX1jXhs6Q88jiz+I7v8UZFG2XJj0lYgbqEjH0KwXMoiyndETkz5rC1kRN3PH3v32GPx9SXgBkuuv5c4fAn5ktqp5MDlnbjoy6WWhjXnTu/K92lv05G8mYX8/GoPBxgDAfnEWwWLQcLxL583vXu2UqdilptAaFp4vKZC2BuCTA8v6R2T6hkb/z3FkHzsAvPnZweHvd8xOM/MPbx99PMmaiIRVDd4ES3YTvgPgAACAASURBVINBV/UcQ3jpb4hQwlGOLP6lev5ht8zpnA211qaazrhsVSv2mBT0o598pxVPvrMzsG17pxNxyvVx6hImknEzzwqSbaCwHHUrT/zdyL/Gj/wB4AjJa5dFl7N9WOz5d8BRbG/agmWLyKibxd9UxF8dhN7VnUEiZkidROkxpOz5F9zPLBb5O/9zR6dOOkvGTMSLtCuTswPtSBXI9lEHmNUBXwD46bkHRR4rTPyjPlvMJCDrzAOoS8QCs32HEi3+YwjWgYHkwI8G5AFUdWxDXVmJCRsIDMv4USP/Pz+/AQuVMrof//Xzea/b2Z3G755ehz8848/erYmbSMaMwN0AEKxjHzaZjK0ZhgWDRYkFSU1/fOQrx2L9Tn9cQo38eebss2t3RaZ6AvmRv+l5/sG2bu/sRzJmetF3GdrvdUxRC8wz8SJ3CNw2PkdtPcFznYwbRa2lnG0H7ixk8VfTZgvZPqV4/r7tI4l/xOv4/ZIxA8cubvHqNw012vYZQ/BPfTQO+paCnN6pRv5R0/jDIv+wuwZZ/BdNrUfWEt5gbrGUwcv//mbgccoVf7UqpdzphE3osm3hZf4Afmdek/DFgPl/x+yJ0/d3MoEWTKnHifv45Y7Z8z9nqZMeyZH/N/76Gjr6stG2D3v+rmiy/6x2jNs6+wO2TzmRP7ctXsQn584hqqPiOx7umNQZx/IEKsC3a2SyVvAuiMVfLiDHcDNk24e/n5Kyfdxd/uXgWdh3RqPTxqjI3+C7BBNnHjADXz15cdH3Hwg68h9D8I9xKBYkqUYK2T5RHV5vqO0T4vlLr589oRartnfjXTd9kyPtaY0pbOv06+lPrEsE1rplknEz1M+VI/+wzsoWIrDE4dELnEyZlPte8me89PS9817PxE0Dr19+sif6ssi092UK2D5uVK5Uo2Txj5uErCWwvbMfLfVJr5MoQfs8+K6kWOQf8zqg8Oe544lafjMZNwKWTs4WSCgNzebsQAfB38/ikAV4wmr7JGMG+rPRYygy3Fl99pg9ISCwYktnpPjLkX8l0ZH/GMLwLojqifzTOQtnXf8Unlu7q/jORQiIvyL2UR1emF8aNstXFuOGVAwJ07dt2G7pzeTwySPnYnK90xnIqyvJRdKSMSP0wi4W+Tt5/s4+1513EK79yIEA/Ii0L1P699qQinuCJZdN6O7PRQqv7/lLNXgMw+t0WLi7+nNISJF/KZEvwxk4xTx/vgsJG0gG/A5H7UT5MyRjZsCaCTvfWcX22d89T/9x4sL84+WVdxDe9xJ1dxJFpsgktDCLqBJo8R9DsPhnctUT+b+7qxevburAZe7M10IUKmwm18kH8iP/qBm/4ZF//r6W1GHWJGKBAT/bFk4lzXTO6xgAYN6kOu/CP/OAGV4HkIoboRd2Ic+fyPGReZ8DZjblef3qwuelUp+MeVko3elcpFipWT6AYwGxcDZIPngiZngipc5c/uLxC/D54+aHt8XNzima5+++d5SsGhHjEZyumVBq5YQFRKrtM6O5BuuvOsMrby3DHR2fCyH8O7JiYwsq/PsrOOALHflrBkA1Rf4ckZcSHYVNguroy+LIq/6By/++Ii/V8wf3veXl90etwKUu8u28tnCqZ03cDKT65WyBHneBj4ZUzLtoa6TSCtMaU57Vk4qbSMaLiL8SsdbGTTfbx/begzlmYUvg/4HAKaFd/dHiHxbJxwzDi67lc5IMiH/wfb568uK8Oj+Mb/uU5vlH1czh7WrFUT5vTTXxQK0kNfK3bKeURimDtYBk+0iRP3fKZUf+VmHxj0upnpVEi/8YgiJuhYcDIQRuXb4xb4IZR1/FCl8B4b49T9J6aMX2PNvnhif8mbhRs2dDxV+xfb5+26uBFM7ahBmI/C1beLV0GlJx31qIG16p5CmNSW97Khbu+ReyfWqTMdhSeQc5f3//WU1Ye+XpgRTPcuEoNZ2zI4WXvyJZzOImeR2rHEkHIv8SvluGs31KzfOPukHgl6u/Ge50J9UnAoO86jXBrytV/Pn3y6fAFsKLzKNqNkXx+WPn4+yDZuL8I+aGPu9F/iEBxFCixX8MYVD4rXAlcNIb13uPH31rB75x+2u49uFVgf1Y5ErxhbOuXfX2tk5slwZWAUfcOWJvSMa8yV1MlO0TVhFRtX1uc7N6mJpEfuTPE+caUjFPMFIxE7VuJDuxLuHZQal40PP/+imLcfCcZmxo6/XvVFTxT5iBAd+UUlisWOGwYqRCSjGo+JF8cMDXW+JSEtBkzF81zCzD9ogqFqfCbYz2/KNsH+dzTq5PBlf6Us43n/+odEuVsAFf/o6KjV+oNNcm8JOPHBhZeyimB3w15eJl+5S5GtXtL27Czc9tKOs1l//9TXz3Lr8WX1faEeMdXcFp9nybXIp48SLqp/7PkzjsykedbVxwzbI90a5PxfIm9bCYCGWiVG/GwiQlLz5qpi2TjBmojfsXphr5s6ak4iau+fASHL7nRCya2hAYbJQv3H89ZBbmTqpDV38OR131D+/zyNQots9QX/gp6U4kSqwML3VTjvwNz1qRf1cJ05/kNZDIv9h3EPMyiYrYPkonwiI9qS4REHzVCs26v6VS/Xo+JQHPPx6cFDdUcBqstn00JTPQPP+v3fYqvnVn+GpHxeBba3KPfufLm/GD+97ynueLpRRfNJOzPZFl5FQ+tn3qk7FA3XzAF9NrHlqJvb7zgFecrSedy8vbLrawekJa4IPfu1OK/DmiTMUNHDi7GbdccARScdO3fZQBX2cMwH8/LrAmEzcN2LZT4TMZMyIj3oEi20hRYsUdtClFwzHDt33kDisZN6RJXqW3VbafCsEdbOSAb0RmG5ffmNyQjFzpCyjf9iFyyjHbwg8yWJxLfY9S4Q5JLSs91GjxH0P4E1+Gz/PnSF/WKnmhE8/2KUHMfvaP1dj/8ocC23zhEZ61U5+K5dXd4ec27XZslWfXtgFwIn+1Tku6SOG7uGkESjZbsu0j3aqr1owv/kHPvzZhok/KOmrvzXrn5dz3zEYyZmDWhBrYQiCdsyuS4id3ZlGpnhwER+X5R0X+5Qx4so9dTPxZZCPz/N2PoP7WvWqYdYmA7ZPn+Zdp+wDBpSOdVM/KRP78mWZPrCmy5+DQ4j+G8Ad8S4v8N7b14sgfPFr2ceSUzHd39eQ9L6dX8g85akxMfq87X96U97y8tq4c+avwcXifjj7nzqAnk8tbparYQjBx00CNZPvkFNuHBSClDMgl2at1F12R32/jbr/q6Lttvd53dMq+07Dy+6ehqSYOW5SXgVIOsu0Tme0TIuZx0whd60Ae8C1lMJ/hO6JiHTD/LMq1fXhex6T6ZCDb57/ueRPbOvxxpHJtH8C5w/GzffzOv9xUz2JwADNvcn2RPQeHFv8xRFT6WxS3v7gJWzr6i++oINefeXeXMwtWtil6MvJMVmdfOTra1tGPh1Zsc5/326rm7qdzlndbbwsUFn93P96HJ2j1Zay8CP3tbV24YZm/BkBDKvh+cZO8kgqAEvnLtk+EJ5uKG1g8rQH3fulo3PzZwwAAu6TCb+/u6skbCCciWLYzRjLUkaTTpuIDvp7nL4lZTCrvELB95AHfsiL/0myfYpF/lO3DHfvEumC2zxOrWvGNv77mPS7X9gGcyF8u75AKmRGtsmR2c8nvz/As8nnKQjdDjS7vMIbgn2Cp2T4DtRfkapWbdvcGjg04Apy1nKnzfJHLEdw5NzyDDW29WHvl6YGLV+2zWrvSgbuYQuL/zb++jvberHe8djcbqD/rrH3b0pBEa1cayZiBO15ylov+0MEzMaUhhfpkLFAGW7YCauIm+rIW2nqcxb5rE2ZgwFeGBYstn31n+DNrrzvvIPzp2Q34y/KNeOqdnTj74FkAfKE1DXcZxwKF1waDPAYRNeAbNmnL8fzdqqPSF0QUPims1HYUG/DlI0VF/n7RueCPZs+WOqxt7cFEZcDXab+zktsJP16G8w6dA6BM8TfI+43KA76FJqzd9v+OGHCtrbmTawf0ulLR4j+GKLe2z0BFRh6U7XI9VvVC681YaKrxSwPIx9rg1szJWLaX3hmGI/7SnYFlgyi/1jqzanu3b/v0Ooux9Lni//eLjsYbmzvw/XvfxHr3bmVbRz+mNKTyLs64aXgXeUPKWSlr/c4ezJpQAyKSBnyD7eCWqnYQABwwqxkH/GszerMWbntxEya6JSK8FbDIsRRydmUif2cBcaNgnn/YwG3MMLwOOmsLrzNM56wBDfhy3ZyPHrZHwf34rizqnaMszuvOPQjpnO3ZaDIGEbZ19GPT7j786MGVAPIXTi+EQcGVvBIxA//9wf28GkxhJCJKfRQiYRrIWHbegjJDjRb/MYRv+5QWaRQrrhVFR58fJfMAm+qj92ZyaKqJh9o+TDpnF2zrKxvbsXCKX2Qr4y4iEnUx9ecsb/WsjGWjP+v8S8UNTGtKYVpTCjc9tdYT/027+/D8uja09WTwvsXO7NnHVraiPhXzxIePtX5XD2ZNcCIx24v8g+3wthdI0bv0tL3w91e3YIPbBtk3t20By7YrEvkD8O7EJtYmQp/nQXnZLomZhL6sH/k318Ud8c/a3gBxOamek+qTWH/VGSW1FYju6L3IXwk6mmrimD2x1vtbxiDK+82XU5HUifx98TeI8PHDC3diA+EfXzs2rxx4JRh3nv/6nT34yl9eGbVlj9e0dhdte6mRf7mTUxg58ucBNtWv506B2xp2+57OWQXb+p9/fxNbOvyB0hueWIuEWUD8M1agHe19GfRlg54/L3UIAH94Zj2+f+9bsAWw1/RG/PS8g/DfZ+2Lw+ZN9BboZgto/c5eL/uCF0mJsn3iBSK9Ca7wtrpZUnEpn90WqFjkD/hpkFMaw+vDqxOZAMfC6s9ajiVlC2/wPJ2zvZTQSrT3/Utm4IvHL8DXTgkvZ+wP+AZ/d3JbfvbRg7yy1/ycpfzeyrV95EleQ52Oy8yaUBsoxlcpxp34f+22V3HHy5vxysb2kW5K2Wza3YsTfrwMVz/wdujzURdEFOVEPTKdUtpjlyLyjNophEb+WTuyrXtPd2qe71TWZiWiyMi6P2chk7O8CTltPRlkcnZApOWFMdZJC6AkYwYaU3F8/Ii5AWuHxSFj2ZjNkb/b5Dzbp4hVATgplzVx01tzVq6Pw/VmBvq9lMqUkHr1QP6CJQAwsS6O3b0ZL8LmFNj+rDWgqp6lEjcNfPXkxYFicmFtVX8/8l3TlIYUPn3UPOk1+Xeo5dk+FMjzr1AfPWyMO/H3a82McEMGwPqdjlXw+uaO0OcL1fYJK2mrer/XPLjSiw57Mzk8/Ob20ONw5D+9OZUX4TO8iErGS/XMP+EZK1r8Z09womy1Dk9HXxYLp4anwPVlLGQsG1MaHIHf4S6xKIv0dEn8eQlGdR8AmONaB7wuLgDPThBexxD8TMXSE5kJtXGvbZ7n71oKlYz8mamRkb/zv2z7TKxLoq0n492hcUnmdM729q90e8PgQ6p3jurvTG6bQZSXHVRO5G9QMPIvJ8W1GhmH4s9/jb4vblePIxiT6gpHbmq2z23LN2L+t+4L5DkD+eJ1/WOrcY07EPadv63AZ/+wHG9t7cw7Tqfr+U9vqkF3P4u/MuDrRv6e5x9m+2TtyFIUPCkpLCXwwNkTQl/Tn3XmAkxpdM7PVvfz1kje/LSm8IkzajmFzx83HzedvxQn7O2vkMURMzdZFb1SA4sJdQlvla9YwPZxIv9KiymfH5WwyH9SXQJZS6DdnTdxxPxJ2HdGI75+ymIvK2gkxJ+PqaY1q+Ml8l2UQZTXWZTr+fOlZevIf/Thp5CNaDMGBK8apa7hqqJeEL9+ah0A5BVLC4tcuGAap3CqNXQAoDudhWkQJtUnvDuFPM/fjfyzhWyfnBUZ+ftlAPJTAqPWNO3PWkjnbE+kt7njBXJUf+DsZrxn7gQcPCeYf61G/jHTwIn7TA0sN8j7/OaT78FHD5uDqQ3BdnglCYpEhPL3J9fHqbTnz0yujxB/XrBE+v1McNvKdypNNXHc+6X3Yr+ZTX7kPwIRsLqMI6NG/vLdrWnQ4GwfwxF9IQREBT3/4WL8if8oXt18V7cjxGoWwwNvbMX9r2+Vqi8Gf+Bb2vsQRtii52qt8TALKWc7uegNyZgv8nnZPlbg/cIuk0zOjhzwLRT5A8BHD5uTt60v6wz4zmyuARGwzs2okUsbtDQkcdvnjgyswgUUX1VJ3mefGY248kP75wnNXu44RbEFtydI2TZ+tg97/pXL9mGirI6GkCUWuSgel/GIKTYKMLKRv/r7UTsiub1E+fuXPcnLFX5A2z6RENHlRLSZiF5x/50uPXcpEa0mopVEdEql2hAGC165lS8rTSZn48Yn1oRGuq1dafz8sdWeiKui/bn/ewmf/9NLvvgrn40HaFUhDQu6OYLniyIbIr48EakuGUN3fw5CiJAB32AK6B0vb/YqWjK9GQs/f2x1fiPgR9nq+gDMlR/aH19XMkH6XfGvS8YwrTGF1Tu6AYRXR1RzqNXIn5Ejx2JVFr90/AL89fNH4sAiszrlyJ/PsyGJWaXE9L0LJ+fNZpb5jxMX4kvHL8C/uBPQ5LY+6M7IljuG2AjaPt4yjoqHr7ZFLU+tBkblpDsbBqE3Y2HPb90XaMNopdJ5/tcKIa6RNxDRPgDOBbAvgBkAHiGiRUKIga1RVyasm2GiNpL84Zn1uPK+t2EQ4TPv3TPw3C3Pb8CPpTr5UR0XD2a9+O5utHal0aJUNlQ7Fjvkffxp79GZQ5YQMAxCfSrm1tm3I2usyJO4Nrf3Be68bl2+EQ9FDCrXlFAGQL3QezKWs1C3Wyjt1U3OwHjYpCu5cBtQKPL3txdbXCNmGjhkj/DxCJnwyN/535ncU5lqjn/898MKPl+biOErJwc7VBb/2901D4Ji6vw/2LUGBoI/vqVE/kU8f9X2KSfd2SQKZJ+NxOceSkbC9jkLwC1CiLQQYh2A1QAOHa6DsxiWWv9muOC87zCbpUYRA9VfZ/gjvfjubnzw508DAM647knv+X4lc6aQ7SOnOOYdx/WlucxCdzqXJ9K9RewgIFgDSIUFW872mdlcgx/+y/7eY/Xa8xfGNjF7Qq33OKw0rhr5R0X1sRDbZ7A01/q2nef5G35nW+7KUJVEHV8Ks30qbVOFEZXcoNo+8kIzYbZPWVU9DQoETKPc9am4+F9ERK8R0W+IiEOimQA2SvtscrflQUQXENFyIlre2to6JA1ivRuO1a7KgUUxLOrjKPpLxy8AEL1Gryy0HGW/va3L26ZG/laI+HOkzhdFaNqocG0fV0B70rnoVE+lU5DrAhXSDN/zd7NiDMKtnzsCH3mP7/VHea4c+TNhlk6dcp7D7g4A1fMfmohctl7YduCPkslV3vMvB/X3GLB9uC7RCKhgVLaPGo3LA/YQ+ddO2ame0jUzrj1/InqEiN4I+XcWgF8CmA/gQABbAfy43PcXQtwohFgqhFja0jLwxasD7+nm+wxmhu/Ft7yMuZfcOyTtYVj8a+ImbFvgj8+s9/zu3kwOqbiBr5y8GFMbk5H1cLqUGvecifNvhzuCmRf5h9z9pJXIf0t7n/c+jGU7U9vrU37knxfhK6mezM4uP3tIzeGXSSm2z9v/fSpmNgfTNKOyLRzx94tiqXdOALzlF5lSIv+hqrMvF6ZTV8PKVtDzHwhEhF9/Yqn3OCzyHxnbx/m/2HUsn8usLbx5J0x5JZ2DNmYVfU0DYlC/ZiHEiUKI/UL+3SWE2C6EsIQQNoBfwbd2NgOYLb3NLHfbsMB6l7WclaG4tHA5/O2VLUPcKqAv6whsPEZ4fNUOfOeuFbjqfmcmb3c650XZcdPA5nanJo1Kh7Subcwg7HSzg2Y2O0KYF/mHiL834OumwP3k4VUB64hfF7B9+gtE/sr2HV1+umlnf3T9EtXzDxPEqIsvaSqRf4iwq5VBS4n8h0qU5WPnef5VFvkDwHGLp3h/y21jP31EbJ+IbB8V2fPPWXbeHX+8nDx/osA1NK4j/0IQ0XTp4YcAvOH+fTeAc4koSUTzACwE8Hyl2qHiZ/vY+PFDK3HBH1/EP9fsLPiasGyWoaYvw5Gy8JZE5GyV3ozl1VSJmwaeWr0T59zwTMH3q02Y3uDUTFcI+7M2Hnt7B+Zeci9W7+jKK6EMABn3xy3fDnPNfsaynR8+WwK9WQsZ5W6kNxMe+ctr/Hb0RYu/F/lnLcRNCo3yoy69RMzwZuMCQCqR/zMvZGcEtlfAf69P5Uf+LGbpnF11A4lypydHysYIDvj6d0qFVz6TPf+sJYrOCC6EYVDAxtR5/tFcTUSvE9FrAN4H4MsAIIRYAeBWAG8CeADAhZXO9OnPWnjdzfzgWV7ZnPDEtVuxS1R+8/R6LLzsfm+SFTOUHQLP+MxattdB7XKP153OeWJVanZCXTLmTczhKDids3DXK85N1mubOkIHfMM8fhXLdsoCs53SnwlO1qpLmJGTv3ZIJRU6ClQulCP/KF826sJNxJwqnvx0qOevRP5R8z8qYcHIkT8LiGxjVFvkLyN3hl5tnxEQQX8xF1FwIF4+lznbLrqKWyFMJVuoer+l0qhYqqcQ4uMFnrsCwBWVOrbK7S9uwuV3r8DL3z3JE7ysbXuWQjIix5vh5QU37+4LZD/0Za2iA0a/eHw1DpzVjCML1PwGpMg/Z3tZMG1uOYfeTM4TjFIj0a0d/bjw5pcAwPPK01npM8fMgrZP2HOMJZwLgQW6LxsU/4n1CcnzD76PbPt0SWMJcsVEAN5KWumsFSmGUZFXMmYgbhqY3lSDze19obaPHPl//ZTFkasmDfUSfUAw8me4k8nk7Kry/FViIXcBI5Ln714Glu0spN6F8AAuIP6W8Gyfmz9zGDZFTH6MPiYFxqmq+GsqierJKasgO7rSyNkC3emcZ3XkXM8fKJ7uxYOBGSt4g9JfIFWRufqBlfjoTc8V3Y9tkpwt0OuKYpsX+VveAKVcLtiyRUlZS5PqEkjEDKzY0uFVskzGjPBsH/f9Ct0B2LaT5y+LvxzhN9XEC3j+wSqdzJ6K+CZjJUT+ERcf2wAzJ9TANCj0bqlOSvW88H0LIjuSSghbQzK/UqVcpbKaI3+1UJq6bbiQ/fZCkX9gwNfy56PsP6sJ5yydHfWy8Pci8hIiAJ3nPyrgKpT9WTuQ7cODN8VKPnDnoGan9BVdhLr0uQRyOYQeyf/PWTZ60znUu5OS5NS1qx94Gyu25BdeU4mZBlIxA4+8tcNL/ZSnqcvwZ7IKLLKSs22YREi50XNfnu0Ti/T8W6PEv0UVf17uL3rlqfcvmYFjFuVngfH3NXtCLVIxI1TYa5OlpW1WQojDBpdlG6Oa8vxV5LERvxz1yKV6Om2KPr783eds4c0ILifFUz6m9vxHGVyFsi9jebXYs5bwLBA1Vzids3DNgyu9iUo8szN/IlNh8Zf339UdLnqM5/nn/MgfANp6M+hJ57xJSfKP9oYn1uIsdzJXFOcdOtv9DMpEMcsOtXZ6MxZsW+TNIpbTQi3buRBqpBIMcgpdXTIWWeq5PaRQHAAsmtoQeMxRVX/WirS6GlNx/OHT+fMDOfL/xJF74Ftn7B362roSl8irhLCFiYasRdUc+cttG0nxl09hqWMOTraP8zsdyDlWI/0q/ppKYnyIP0f+OcuLxnOW7fXiaqmHm5/bgOsfW43/fXwNAD8KZSuDKRb590gi/kaBCF0IIZVGtgMzX3vSFnoyluf5F1olKowrP+TMiFWtrXQ2XPwBJ3tHzYrolj67LYRrpxiIm+R5/vtMb8S/Hz0PB8xqQjpnB84xs1sZ5J0zsRZrrjw9kE4I+Bd0zhZFB7mfvfQE3PmFI73HLP4HzGrGxyLWig2b9RtGpRdWYeQOoao9fzNE/Edikpd0zFIPn7WcrD2igZ1j9SU61XMUwGWK+93aLwDbPo4wqbP+eDKUPDgK5GcF9RWJ/OU7g827oweXspbwvPGsbXt3HIDTgfTI2T5l/GgTpm95qBF4xrIjbalcyF2BPIHMcj1/wMmk6cs4i7IsmlqP75y5j9dR9WTyOxE1vbM2YcI0CIfsMQGPfe04b7t8cRa7RZ/WlMJBc/yaOqWUYeD2NxYodAZUZsA3DFnMqjHy5+8jLNtnJGv7AKULec52PP+4EW4FFkPt5KrwayqLcSH+LFx9WUsSWcn2UQRKIPiYxUSd6SqL/5tbOjH3knuxkj11WwSWCVSXI4x6n2xOeOUcAGfQNyetnVpMCL960iKcsJcTRcv5z+pdSjprhQ74AnDrygc7C3ndXsst6Qw4EXRf1kJWGpjltvZmcpFF6PicynaUnHFjBDzd0n6mnNVUakR282cOwwMXH1Nwn+ESYvkGwxymDqcc+M4xbNLbSE7yAkr/vjnbZ6BrV6udnPb8RwGd0oAv2xA5acBXFTpPE93vlq0WFmX+DciCet/rWwH4pW+/dMvLOP83/ty1guIvvU/WCkb+vAAL16IpFokmYoY3ECv/yNXSyOmc7ZV0PmXfqYHnbJE/GYbHTQB3hq/7w69JmJ7nz+eJ71J60laktcT71CiDn8ctdgZw5Sir1Iv1z589HOcfsQf2mBSetqly5ILJmNEcvrIXM1wWTCCSrUJR4UAiHmb7jGB5B+fv0o6ftZ071HLKOMvkR/7V9z2Vw/gQf9dqkFMSs1Kqp1orh+0Q/nL5MXv+/OORI3ZvFSQhsLsng3te2xp4T16IJQxZ7Nnz5+Jfnvi70XSxtFTTIM8aku8S1LubTM6ZTJaKG/jCcQsCz8kDvgfMagKgRP5CeJFqTdx01s7NWV7beDC1N5OLFH/OdVcnYP3uU4di/VVnBCLhUiPLOZNq8V9n7TekYlQpz3+/mY2Bx0aV2z4s/mGpniMhguWU3TjWzQjLWQLZEsaQSjkmoG2fqsd28/sBoE+yIWTPX10QwluOz33Mwsnvw+IqR+z8Q7BtgWfW7gq8X0MyhlYl8v/eXW/gcoTLUwAAHrxJREFUK7e+gpOvXRYon5C1nGwfXopwTatjHfGi28Uif9Mgr3MqNO09nbOd0sxEee9pCye6P2SPCfifjxwIIN/z5wsh5do+Gel2mtMoe9LR1tJcNzqPGngNeOADjNSGgkpFtXddeDTeueK00ONUY6ond+zy18m/m5Es6ez8XXjf33/6UJx36Bxs7ejHspWtA0rzBMKyfUa3+lffr2yI6cn4E7vkwUbZ1uC/MzkbrV1pz/Hn75YnUnH2DouRPKArL379GpeScJkzqTbP9vn9M+/ijpc2Y9X2btz8/AZvO0f+U9z1YXkBda5VU2zCU8zNwgEK3yVkLNuJ4InyolvLrWVkGuTdcciZTpaUi86Rfzpne1F8veT5h1UOBYC9pjmpnVFrE8hiWE7N9aGmUsJmSt8TEBSw4RpkLgde9lIWwGoZ8C3l+Pw9bm7vG/D5VV82yrV/7It/pxSxyn/LM085E+YPz6zHydcukyJ/zpRxI3/39fxDevSt7Zh7yb14Z3uXJ1aWDby2qT1Qs32PSbXYGTG5CQhOfGLPv8WN/N/e1gUiYEaz0xlEiX+ju66vaRheBB4W+bO/n85a3kzdvMjfFrDc22O+yGQRt4XwLoSahInO/iyE8AdxeU5Cdzp6wHfxNMf22BwxxT444DtyV5k5TNHtQLJXhpPrzjsIv/zYwYGy2pwFV2ra7FBSrucfSFEdoGrryH+U0SlF+3IhMXk7i/v2zn7s7s16/jZ/t9w5sO3Dgrb83d0AgGWrWgPjAyu2dAZmns6ZWIfO/lzo+rwA8M52f8GVrGWjJ22hPhXzBnmnNqS8Cy1KGBpTLP6+Tx3WUdzw8aWY0pD0In/ToDxhE8KfaerZWZKGy7ZPTdxEu3teOfL3Zudm7cjIf7E7qSsqBTasdPBI4BUvq7AgB2asVqH4N9XEcdr+04PbauP4/acPxQcOnDHs7TFD7kAKIV8LxSZnRh5zjA34VnoN3xGnKxD5+4K/W5ppyrYOjwHwRCT+alnsu9M53LBsTV6uentvFo01MW/fjr4s5rfUe89zSeX23iymNoas1OX+GBtTMcfzz+RQEzedmbIZC7Mn+tFWWCVOwF8dyiDyavHLA1uPfOUYrN/pjC0kYgbSWRuphOnYPma+7ZOznRozfJHJx1U9fz6XyViw08nadrTnP9mxsU7bf1ro8/KFNdABuqGAI8Yj50+q6HGqPfKP4tiQ8hrDARGByAlUStFguUNVU7ZLRRX7UfQ1hTLmxV+O8Nk/B+AtdAI4A7dH//Af3o+C/Xl/4RenU1ixpTO0ls7u3oy3Litn7sjLBKZcUWR/O2pyVVNt3C0+5dQor0/GsKMrjdnSqlRR5YI48reF8BaokG2fBVMasGCKE20nYwbSlo2E7UT2aqTJqZ4xw6+jHxn5J4y8yXAsmDkrv0wEk4yZeOW7J+WVVmaCtVtGLvJPxkw8ePExgQ64EhhVHvlXI7ysomkQvnHq4oKReGwIIv+xluc/9sVfsnDW7+pFS0MSE2sT2NDmZ9hs70xjk2Q/cHolZwEVq9vf3pf1fgg8F0BeOpDF0Fs8PqJiZmMqjkzOnYVoGl7J431nNnn7RKVOcuQvF0KLsksSMRPprI3aeLjtc8KPlwFwiq3JWUxeG9yBYiDo93INJO58sla07RMzCM21idDngIHl+VeKxdMaiu80SOSvoBqzfaoRkwgWnN+imq6sUs7M+MjjKV/LaO+jx/yvjCN/jozPPmgmGlKxQJpmdzpo43DZYc7/L7ZU3OubOrBpt9OZcEaQnL/OIpxzJ5k88tZ277n5UjXLxlTca1fCJG8g+JA9/NIFUbYPD/j2S2sMRAUmyZhRMNuHMQ2jqO0TEH818rdFqO1DVDxDQ36+nKX2RivVXt6hGuFTVkq2T4HlKUpGXed5tHv+Y/6qkks7AE71SDV7RC02xnX0/dr2hSP/DW29+O3T6wH4KZE1cRN7THLsGr6Yc7bAZXe+ji/8yVlk5dhFLbjoeD9iqUua3sQx+TZ1n+n+hKCoH7EX+Wf9fPtCi52ksxaEcMoKRGXTxA0p2yfC9kkl8iN/fr9szg61qUoVc698QBWmPg418nc12uvEDxd+Ybni+/ZmB+bzy6hZTaM9JhkXtg/XnwGA+VPqAyP/yZgRWWZYXthEXWkqCs/2iZt46MvHwLaBp1Y7awTnLIFbl2/y9n3/khlY0OJbCjHD8DqPuGng2o8swfqdvQHvPspG4TubdM72fPSoayIRM9CdznllGqLE1TTIi6448r/rlc3Y3tnvRarykoS8YhaLu1oCW37fUuDb+oFOyhlNVHu2TzVSzgzjngEO8srUKOs+a8+/yunsy6Gxxrd55rfUeWLi+M7xvMifyUiR/7TGVGROugynidYkTN8GcS/mXcoawKm4gUn1vu8djxneYFTCJHzooFl576/aPh9YMgN3v7oFc9xJYM21cSnyD29jMmZiV3fGLdMQbfvETPJE3rYFtnf24z9ueQWAL1a1Ul18jvwNg2BQfj0h+X1LwTAAWONDDIOe/9j/vEMBn6ZS7pR60wMb5JVRS5GM9m9pzIt/VzqLxlQc292FwxtScU9MmmvjiBlGYF1ZmZfe3Y1/rt6JnC0wrSla/Oe31HllGDgNNGzAd3tH8DipmBlYEzhuUqjtI6N66Ff/6wH40YcPQMI0kIgZOG2/abj9RefuIurHmYwZSLtrG5hEkWITM4yA7SPXJ+ILrj4pe/5+m2OmERn5lyrm3PGMZLbPcKGzfcqHz1kpp0ueof7JI+cO6HjqCmxR42+jhTEv/p19OTSkYrjrwqPQ5to7HPk31cRh2SIy+2b9rl589KbnMGtCDaY1pUL3qUuYWDClPl/8pSiBxbW9T438zUA0kTANLzUyyupQf28J0/AugvcvcSbbxLwB32jPn1fyKnTLLNs+lhABe4wFSk7VlD9L3KC8SW2puIH+rF1yNgt/rsQ48PxHa57/SFLOBLwDZjXjwRXb8cDF78Ve0xqL7h+G6vkXSwSpdsa++PdnMbEugSWzm71tbIs01yYC8wCi2NGZRn3Esn+JmIGElAXAlrz8Q2EhV+0lNZKQ7ZCo9Eau8cOE3fJ6tk/oO/iTvCzbf/23Tt8LOVvg6gdW+u1x8/ydyTQiUJyOxUoWfzXy50VxeDJOKm6iP2uXXBPFH/Ad+5F/sJCdFv9S4OCmFO/9c8fOx6n7TQtMviwXVfxLGQOsZsb8VdXZl/UGQxlOi+zNWCUJS8ayEY8Rvihl5ngra5lGqFDLto8X+ffmR/5MQzIWiPajipl98fgFuOn8pQXbW2qqp7Mco7PtgmPmY8ms5sB+fG5MIjz61g7P75c/U31A/OUOz4/8Wdi4XaXeLfu2z9gXw8CatKM9jWSY4N9uKeUdTIMGJfxAMLMNyLdgRxtj/lfW1Z/zSi8wXzhuAfaYVItT9p1a8gSimGHgqycvxjS3tLK8oHrYsoFBC8SN/HvUyN/ZZ8V/noLnLjshIPhRnVLcNHDC3lNCn2P82+Dwz5ZKmOhNO0taFloLNeZ5qoQ3t3aGPlcbkurpPB+M/IHyq3PyXcm4yPPXnn/Z+Nk+w3M8HflLENGHiWgFEdlEtFR57lIiWk1EK4noFGn7qe621UR0yWCOXwqWEHmR/7SmFB7/2nG4+MRFJacRcifBUWhd0i9iFiZqYZ7/7rzI31/2sDYRjPwLdUrFbnM5IIm6KPacXIeMZePdXT3B/HLlfbndYYczQiL/VCw4yM2RPx/DT1kt7aIZT5F/OYuTaBw88R+m8zXWxH+wnv8bAM4GcIO8kYj2AXAugH0BzADwCBEtcp/+OYCTAGwC8AIR3S2EeHOQ7Yjkle+eHFpLhwVJjbKcTJj8LBU5PRTwV6uKu1k2Mk6t9nz/vj3P8w/+mGSRG1wNe+fzRvURPOD17q5eTK5PettV0eFshjAxMkM8/+Bn9s+jb/uUd5HyccdDnr9O9SwfowzbZyhQr9eoulWjhUFdVUKIt4QQK0OeOgvALUKItBBiHYDVAA51/60WQqwVQmQA3OLuW1EKRcqqsERl9bANw/9zxBszKU/8a+Nm4Jgxb8A3ExjkVSMJ+X3iBVbhKkYxK3LR1AZPbMxA5B/cjwvRhWUEGSHCHPjMBnm2D79vuSLOF/dI1/YZDnS2T/l4i8kMk/ir12vUhMvRQqVCqpkANkqPN7nboraHQkQXENFyIlre2tpakYaqlsLUhnDx58JQntedzM/mYdSBIX5Ne28WDZIFpUYSNYF6QAP/Qdue7RP+HjUJ01tGUbbT1f15klvY2xRrn5PnzwveD3LAdxx4/tW+hm81Mty2TyqRX/p8NFP0qiKiR4jojZB/FY/YhRA3CiGWCiGWtrRUpm64KtxTGpPh+ym16tn2ESJ/xSw1QuAOJmPZaJBsEjXCS4Wkhw4EUcT2AeDZPUYBz5/LW3A7Z0+s8ZZfLBadxk1CeogGfLXnrwmjnEleQ0EwfZvwvsWFEy+qnaKevxDixAG872YAs6XHs9xtKLB9RFD9Ql4onYmbhKxb2x7whShewJ9XxV++mOtT0adcfl2hxdeLoS5DGQZn5shtyxN/t6opbzeJUGo9lZg0ycuziNxFZkqNl9QU0bGMfDrrI9Y40AQZrlXWGDk4e+eK04flmJWkUlfV3QDOJaIkEc0DsBDA8wBeALCQiOYRUQLOoPDdFWpDSXAFT/79yAOgANCiRMicdsiCJoC8VE/V9pFTFRtSMfzpM4fhaycvgkoqkCo58B/0MQtbsGhqPb50wsLIfbjNwYWwg/tw5C/XUDEV+yuKmGl4kb9q+5TKeBrwlQVsjjKRTxOOV9J5mDz/sfY7HGyq54eIaBOAIwDcS0QPAoAQYgWAWwG8CeABABcKISwhRA7ARQAeBPAWgFvdfUeM7W5dH545WxM3cNAcf7JTi3snwIXh/DKyrvgLkRelHz5vYuCxKd0l1CdjOGrBZFx0fL4wJ4fI9mmqjeOhLx9bcBESnpAla3i05y9F/nyrXYrtkwt2HuV699ye8WD7yOd+tFeLHC4SIQGMpnQGm+1zpxBilhAiKYSYKoQ4RXruCiHEfCHEYiHE/dL2+4QQi9znrhjM8YcCXrVr1gR/mb47v3AU9nQXWeHIn0vCfuqouQCA/aTVtWSh/tdDZuHLJwWjejnyj1q2EBg626cUOPIv5DVnlFRNg8irnS7r8ZPfeB8e/eqxgdfGDCOv80jE/A6zFLzIfxwM+LJ1eNI+U0e4JaOHJnemvh4iGRjj3lzkdMRZzbUAdnl+NFsePADMpZZP3nca1l91Bh5+01mNSx7wnTWhBtd8eEneMWRRVTN8ZMoZ8H320hMGFRF75ZcLpHp+eOnswHbZ9pE/k1pvCAiOiXDnoQd8o5lQl8DtnzsC+89qKr6zBgC8ZUD1APnAGPshVRG+ctIiNNXEvVo8HJTyYOdkJfJnWPAFhCdqUQGt7I+HlYJgAqmeRQRvWlMqb3yiHNj2iRrwvebDS7wo1LN9DGnwt0g0Lls8ap5/6QO+/LrxcXEvnTsxb6lATTQTat3IX4v/gBj34v+lExbi1e+d7D1WI/8zD5iO6U0pfProeYHXsYjLkX9UfW9e3AQoFvkXL+w2VIQO+AYmGkH62/f8/ci/8PvLk9S82dQDHPAdD3n+mvJprokX32mIefjLx+Cpb75v2I9bCca97cOw5z/ZXVnrx+cswXWPvoP5LfV45tIT8vaXPfli4g84wmoLUXLkX+nMAk/8IyL/MDuIJPEvNsgWl96X76rUFNhiDDRLSDM+YNunu3/wSzSWysKp0UkUow0t/i6fOmoeZjbX4NT9pgEAjls8BccVmMSRkCwM/rvQbG+eDVjotj4ZUgyuUvCx5CnqRki0D8hWT7jnH4ZsW112+t54du0unHHAdPzm6XWlz/D1Uj31bb0mn2bX9unsL74mhyYfHVK5mAbhtP2nl5xm59s+fqpnIVHj59QFXGTKjYwHA3dYbG8Biu0TyP+XbJ8SJ9bIFs/EugQuPX3vgpZXGONpMRdN+XC13lIWZNLko6+qAcKdhIA0+FtCSFso8h/OCJezfWTxlwWdQmwfw5AHf0u3fdSJYeWmeupaN5owmrzIf/hsn7GEFv8B4mmj8KPoUgo9FYr8h3NyD9+5yGVpg6tJ5ds+BlFggZdCyNH6gGf4as9fUwCO/Dt05D8g9FU1QLiw28Kp9f6AbwklXqsllY/bEYj8I7J9wjz/QoPbQNDz57/LHccwtOevKcCEOkf8GwvUy9JEo8/aAJnWlMKfPnMYlsxuRq87B6AUN6NQts9wwu3IWtKAb1S2j+Fv483FVjGKG/mRP3cC5RZ2056/JowpDSn85JwlOHrB5JFuyqhEi/8gOMr90eXc6LkUUSt30LNSsOefixrwjcj2YdunmPjLkX9emQad7aMZIs4+eNZIN2HUokOqISDmpXqWYPsU8PyHE9/2CU/1DCs0ZpBvxRSN/OXF6EssBqfilXfQk7w0miFHX1VDAKdoXnLaXkX3rTbbJ2eHR/7BtE9/G1sxRT1/+c5BmRswyZ1IV4zxVt5BoxlOtO0zBJgGYf1VZ5S0b9XYPiGRf1SFz7ABXyt/jfsAsZDIvz4Zww/O3h/vXViaR2u4NpMucazRDD1a/IeZYpH/Q18+xiulXNF2hOT5B1M9/b/ltVI926dI5C9H6/JdxHmHzim5jSbRuKjoqdGMBFr8h5likf+iYaod4tk+ZWb7eAXnito+fu8x0FIVpkHjopa/RjMSaPEfZqrH83c6IdnzN6PE31vJC7j4xIVI5yz86yHyUsz5hGX7lItpUKA6qEajGTq0+A8z1TLJiyemyRZT0Rm+BqG5NoEfnH1A0fevGYIidecdOgdL504Y0Gs1Gk1htPgPM9US+bMnHyzvEGX7+IXdSqVRqrVezutk9pvZFFguU6PRDB3VoUTjiGpZdajWLU/xjVMWhz4fjPz5/zLEX5pyr5fZ02iqDx35j1OKpadGZfuUSkNKivy1+Gs0VYeO/DWhhFlA5ZTYaazx4wqt/RpN9aHFf5ioFq+/VIKZP/nbitEoRf56kpZGU31o22eYePKb70Nn3+hZdCIs26ccER9tnZ1GM94Y1BVKRB8mohVEZBPRUmn7XCLqI6JX3H//Kz13CBG9TkSrieg6Gidh4ZSGFBZMqR/pZpRMYGF391dSjnc/Tr5WjWbUMtjw7A0AZwN4IuS5NUKIA91/n5O2/xLAZwEsdP+dOsg2aCpA2IQvPXCr0YwdBiX+Qoi3hBArS92fiKYDaBRCPCuchVz/AOCDg2mDpjLIOi8v46jRaMYGlTRm5xHRy0S0jIje626bCWCTtM8md1soRHQBES0nouWtra0VbKpGxQjN8x+hxmg0miGn6IAvET0CYFrIU5cJIe6KeNlWAHOEELuI6BAAfyOifcttnBDiRgA3AsDSpUtLXf1PMwSYYTN8tfprNGOGouIvhDix3DcVQqQBpN2/XySiNQAWAdgMQF53bZa7TVNlyJE/YWC2z56T67B2Z8+Qtkuj0QwNFUn1JKIWAG1CCIuI9oQzsLtWCNFGRJ1EdDiA5wCcD+BnlWiDZnCERfnlRv5//+LR6MmMnvRWjWY8MdhUzw8R0SYARwC4l4gedJ86BsBrRPQKgNsBfE4I0eY+9wUANwFYDWANgPsH0wZNZQjT+XJdn7pkDFMaUkPTII1GM6QMKvIXQtwJ4M6Q7X8F8NeI1ywHsN9gjqupPLLFI+AMt1RLUTqNRjN49DRMTSgBi8cdah9oaWaNRlN9aPHXhBIm9DrbR6MZO2jx14QSZvHokg0azdhBi7+mKDzBwtTar9GMGbT4a0pG2z4azdhBi7+mZHS2j0YzdtDirymKU4NPF3bTaMYSWvw1JaNTPTWasYMWf01ReMBX2z4azdhBi7+mZLT2azRjBy3+mpLRnr9GM3bQ4q8pitArKWg0Yw4t/pqisPbrwF+jGTto8ddoNJpxiBZ/jUajGYdo8dcURWjTX6MZc2jx1xTF9/y16a/RjBW0+Gs0Gs04RIu/JsCek+tGugkajWYYGNQavpqxx4NfPga26vG7D7Xpo9GMHbT4awLETX0zqNGMB/SVrtFoNOMQLf6aogjX99HJPhrN2EGLv6YowvP8tfprNGOFQYk/Ef2IiN4moteI6E4iapaeu5SIVhPRSiI6Rdp+qrttNRFdMpjjazQajWZgDDbyfxjAfkKIAwCsAnApABDRPgDOBbAvgFMB/IKITCIyAfwcwGkA9gFwnruvRqPRaIaRQYm/EOIhIUTOffgsgFnu32cBuEUIkRZCrAOwGsCh7r/VQoi1QogMgFvcfTVVjGf7aNdHoxkzDKXn/2kA97t/zwSwUXpuk7stansoRHQBES0nouWtra1D2FRNOSTjzs/E1Et5aTRjhqJ5/kT0CIBpIU9dJoS4y93nMgA5AH8aysYJIW4EcCMALF26VFcXGyEuf/++mNFcgxP3njrSTdFoNENEUfEXQpxY6Hki+iSAMwGcIPzyj5sBzJZ2m+VuQ4HtmiplQl0C3zx1r5FuhkajGUIGm+1zKoBvAPiAEKJXeupuAOcSUZKI5gFYCOB5AC8AWEhE84goAWdQ+O7BtEGj0Wg05TPY8g7XA0gCeNgt9/usEOJzQogVRHQrgDfh2EEXCiEs/P/2zi5EqjKM478/ohYpmR+EuJIaQkiELVsYiYRRqUVb4IUQ5EUQVEIRQYoQdtFFQZ8QSR+mfWpZkXhTlkJXra256pqpWxol5vaBVjf24dPF+4yehpmtzZ1zZuc8PxjmPe85y/ub/5x5ds57zu4BJC0DPgBGAGvMbO9ZOgRBEASDRMPlRh0dHR3W3d1dtEYQBMGwQdIOM+uotS7+wjcIgqCERPEPgiAoIVH8gyAISkgU/yAIghISxT8IgqCEDJurfST9AHzzP398IvDjEOoMFeE1OMJrcDSrFzSvW6t5XWRmk2qtGDbF/2yQ1F3vcqciCa/BEV6Do1m9oHndyuQV0z5BEAQlJIp/EARBCSlL8X++aIE6hNfgCK/B0axe0LxupfEqxZx/EARB8E/K8s0/CIIgyBDFPwiCoIS0dPGXtEDSfkl9kpYX7HJY0h5JPZK6vW+8pC2SDvrzBTm5rJHUL6k301fTRYlnPMPdktpz9lol6Yjn1iNpUWbdCvfaL+mGBnpNlbRN0heS9kq61/sLzWwAr0Izk3SOpO2SdrnXw94/XVKXj7/B7+mB3/djg/d3SZqWs9daSYcyec32/tz2fR9vhKSdkjb7cmPzMrOWfJDuF/AVMAMYBewCZhXocxiYWNX3GLDc28uBR3NymQe0A73/5gIsIt2bWcAcoCtnr1XAAzW2neXv6Whgur/XIxrkNRlo9/ZY4ICPX2hmA3gVmpm/7jHeHgl0eQ5vAUu8fzVwl7fvBlZ7ewmwoUF51fNaCyyusX1u+76Pdz/wBrDZlxuaVyt/878S6DOzr83sd2A90FmwUzWdwDpvrwNuyWNQM/sE+Pk/unQCr1jiU2CcpMk5etWjE1hvZifN7BDQR3rPG+F11Mw+9/avwD5gCgVnNoBXPXLJzF/3b7440h8GzAc2en91XpUcNwLXSunuUDl51SO3fV9SG3Aj8KIviwbn1crFfwrwbWb5Owb+YDQaAz6UtEPSnd53oZkd9fb3QJF3SK/n0gw5LvPD7jWZqbFCvPwQ+3LSt8amyazKCwrOzKcweoB+YAvpKOO4mf1ZY+zTXr7+BDAhDy8zq+T1iOf1pKTR1V41nIeap0i3xD3lyxNocF6tXPybjblm1g4sBO6RNC+70tIxXFNcd9tMLsBzwMXAbOAo8HhRIpLGAO8A95nZL9l1RWZWw6vwzMzsLzObDbSRji4uyduhFtVeki4FVpD8rgDGAw/m6STpJqDfzHbkOW4rF/8jwNTMcpv3FYKZHfHnfuA90gfiWOUw0p/7i/IbwKXQHM3smH9gTwEvcGaaIlcvSSNJBfZ1M3vXuwvPrJZXs2TmLseBbcBVpGmTyn3Ds2Of9vL15wM/5eS1wKfPzMxOAi+Tf15XAzdLOkyanp4PPE2D82rl4v8ZMNPPmI8inRjZVISIpPMkja20geuBXvdZ6pstBd4vws+p57IJuN2vfJgDnMhMdTScqjnWW0m5VbyW+JUP04GZwPYGOQh4CdhnZk9kVhWaWT2vojOTNEnSOG+fC1xHOh+xDVjsm1XnVclxMbDVj6Ty8Poy8wtcpHn1bF4Nfx/NbIWZtZnZNFKd2mpmt9HovIbybHWzPUhn6w+Q5htXFugxg3SVxS5gb8WFNE/3MXAQ+AgYn5PPm6TpgD9Ic4l31HMhXenwrGe4B+jI2etVH3e37/STM9uvdK/9wMIGes0lTensBnr8sajozAbwKjQz4DJgp4/fCzyU+RxsJ51ofhsY7f3n+HKfr5+Rs9dWz6sXeI0zVwTltu9nHK/hzNU+Dc0r/r1DEARBCWnlaZ8gCIKgDlH8gyAISkgU/yAIghISxT8IgqCERPEPgiAoIVH8gyAISkgU/yAIghLyN2Jgihe2pGj6AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mV5jj4dThz0Y",
        "colab_type": "text"
      },
      "source": [
        "另外，`avg_final_reward` 代表的是多個回合的平均 final rewards，而 final reward 即是 agent 在單一回合中拿到的最後一個 reward。\n",
        "如果同學們還記得環境給予登月小艇 reward 的方式，便會知道，不論**回合的最後**小艇是不幸墜毀、飛出畫面、或是靜止在地面上，都會受到額外地獎勵或處罰。\n",
        "也因此，final reward 可被用來觀察 agent 的「著地」是否順利等資訊。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txDZ5vlGWz5w",
        "colab_type": "code",
        "outputId": "ffbecb9d-e408-4cbc-8bed-4c1bddffc8c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "plt.plot(avg_final_rewards)\n",
        "plt.title(\"Final Rewards\")\n",
        "plt.show()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2de7wcVZXvf6u7zzmBBAgxIQQCBDSIARExoHd8IwioMziOOsxTHWcYHXTUcS4joo5eh891mKsOjvhg5OEDFFQe8TEoQR7ySEISXgkQCJCQhDxOEvI4eZzTj3X/qL2rdu3au7q6z6nqPqfX9/M5n+6u2lW1q/r0qlW/tfbaxMwQBEEQeotSpzsgCIIgFI8Yf0EQhB5EjL8gCEIPIsZfEAShBxHjLwiC0IOI8RcEQehBxPgL4xIiGiKi48ZgP18koh+NRZ86BRHdRUR/2+l+COMLMf5CV0NEa4honzL2+u8IZp7CzM/mfOy3EFFDHXM3Ea0iog/leUxBKAox/sJ44A+Vsdd/LxR47BeYeQqAgwF8CsB/E9HLCzx+CAXIb1YYE+QfSRiXEBET0cvU+2uJ6Aoi+pXy0BcT0UuNtpcT0Toi2kVEy4joja0ejwN+DWA7gJPVfktE9BkieoaIthHRjUQ0Ta37PhF9Wr0/UvX3QvX5pUS0XW1/KBH9kogGiehF9X620fe7iOhSIroPwF4AxxHRWUT0JBHtJKJvAiCj/cuI6G61bisR3dDG5RV6ADH+wkThfABfAnAogNUALjXWPQjgFADTAFwP4KdENKmVnStD/UcApqv9A8DHAbwbwJsBHAHgRQBXqHV3A3iLev9mAM8CeJPx+ffM3EDwG7wGwDEAjgawD8A3rcP/FYALABwEYCeAmwB8TvXlGQCvN9p+GcBv1XWYDeC/WjlPoXcQ4y+MB24hoh3q7xZPm5uZeQkz1wBch8DYAwCY+UfMvI2Za8z8VQADALJKN0cQ0Q4ERvlmAP/EzA+pdR8BcAkzr2fmYQBfBPBeIqogMP5vUDLNmwBchshIv1mth+rXz5l5LzPvRnDTerPVh2uZeaU6t3MBrGTmnzFzFcB/AthktK0iuJEcwcz7mfnejOcp9Bhi/IXxwLuZear6e7enjWkA9wKYoj8Q0T8T0RNKCtkB4BAEXnMWXmDmqQg0/28AOMNYdwyAm/WNCcATAOoAZjLzMwD2ILgJvRHALwG8oOIFofEnogOJ6LtEtJaIdgG4B8BUIiobx1lnvD/C/MxBZUZz/UUIZKAlRLSSiP4m43kKPYYYf2FCo/T9iwC8H8ChypDvhKGTZ0F59v8C4JVEpG9A6wCca9yYpjLzJGbeoNbfDeC9APrVsrsBfACBJPOwavNpBE8hr2XmgxFJQ2b/zNK7GwEcZZwfmZ+ZeRMz/x0zHwHg7wF8S8dGBMFEjL8w0TkIQA3AIIAKEX0BgRffMsw8AuCrAL6gFn0HwKVEdAwAENEMIjrP2ORuAB9D4M0DwF3q873MXDf6tw/ADhUs/tcm3fgVgBOJ6D1KXvpHAIfrlUT0PiNg/CKCG0ej1XMVJj5i/IWJzm8A3AbgKQBrAexHXCZplasBHE1EfwjgcgALAPyWiHYDWATgtUbbuxEYd2387wVwoPEZCDT7AwBsVdvflnZwZt4K4H0AvgJgG4C5AO4zmpwGYDERDam+fSLv8RDC+IRkMhdBEITeQzx/QRCEHkSMvyAIQg8ixl8QBKEHEeMvCILQg1Q63YEsTJ8+nefMmdPpbgiCIIwrli1btpWZZ7jWjQvjP2fOHCxdurTT3RAEQRhXENFa3zqRfQRBEHoQMf6CIAg9iBh/QRCEHkSMvyAIQg8ixl8QBKEHydX4E9FRRHQnET2uaot/Qi2fRkS3E9HT6vXQPPshCIIgxMnb868B+DQzzwPwOgAXEtE8AJ8BcAczzwVwh/osCIIgFESuxp+ZNzLzcvV+N4KZjo4EcB6A76tm30cwD6ogCMK4ZMlz2/H05t2d7kZLFKb5E9EcAK8GsBjBNHcb1apNAGY62l9AREuJaOng4GBR3RQEQWiZ93/3AZz19XuaN+wiCjH+RDQFwM8BfJKZd5nr1BykiUkFmPlKZp7PzPNnzHCOThYEQRDaJHfjT0R9CAz/dcx8k1q8mYhmqfWzAGzJux+CIAhCRN7ZPgTgKgBPMPPXjFULEExkDfV6a579EARBEOLkXdjt9QD+CsBjRPSwWvZZBPOP3khEH0Ywr+r7c+6HIAiCYJCr8WfmewGQZ/Xb8jy2IAiC4EdG+AqCIPQgYvwFQRB6EDH+giAIPYgYf0EQhB5EjL8gCEIPIsZfEAShBxHjLwiC0IOI8RcEQehBxPgLgiD0IGL8BUEQehAx/oIgCD2IGH9BEIQeRIy/IAhCDyLGXxAEoQcR4y8IgtCDiPEXBEHoQcT4C4Ig9CBi/AVBEHoQMf6CIAg9iBh/QRCEHkSMvyAIQg8ixl8QBGEUMHOnu9AWYvwFQRBGwTi1/WL8BUEQRsM4tf2dM/5EdA4RrSKi1UT0mU71QxAEYTQ0xqnr3xHjT0RlAFcAOBfAPAB/RkTzOtEXQRCE0TBObX/HPP/TAaxm5meZeQTATwCc16G+CIIgtI14/q1xJIB1xuf1alkIEV1AREuJaOng4GChnRMEQdDsG6njiwtWYmi41umujCldG/Bl5iuZeT4zz58xY0anuyMIQo/ygwfW4Nr71+C7dz/jXD9OHf+OGf8NAI4yPs9WywRBELqK/dVG6nqRfVrjQQBziehYIuoHcD6ABR3qiyAIgpdaIzD+lZLbXI5P0w9UOnFQZq4R0ccA/AZAGcDVzLyyE30RBEFIo1oPzHulTM7149Xz74jxBwBm/jWAX3fq+IIgCFmoh56/2/iPU9vfvQFfQRCEbiDy/D2yzzi1/mL8BUEQUqg3AuPe55F9xqntF+MvCIKQhg74lj2yz3jV/MX4C4IgpFBTsk/fBMv2EeMvCIKQQk3JPuL5C4Ig9BDa+PtSPcer6y/GXxAEIYVavZnmX2Rvxg4x/oIgCCloz9+n7vA4df3F+AuCIKSgPX+fti+evyAIwgSk1sS6yyAvQRCECYhO9fR5/uPU9ovxFwRBSEMP8vJq/mL8BUEQJh7V0PN3r5c8f0EQhAlIPcz28cg+RXZmDBHjLwiCkEK1ni77iOcvCIIwAQk9f4+PP05tvxh/QRCENHSqp0/zl1RPQRCECUjTbJ8C+zKWiPEXBEFIoVmev2j+giAIE5Cwto9n/Ti1/WL8BUEQ0qiF2T4S8BUEQegZmlX1FNlHEARhAtJM8x+viPEXBEFIoVm2z3i9KeRm/InoP4joSSJ6lIhuJqKpxrqLiWg1Ea0iorPz6oMgCMJoqUpVz5a5HcBJzHwygKcAXAwARDQPwPkATgRwDoBvEVE5x34IgtDlLHx8M57avLvT3WgL8fwtmPm3zFxTHxcBmK3enwfgJ8w8zMzPAVgN4PS8+iEIQvdzyS2P4Zr71nS6Gwl+u3JT+N408svWvojbVmwE0HyQ1w0PPo/VW4bCz1t27ce37lqNG5euw2dvfgwbd+4L163eshs3PrgOAPD8tr344aK1Y3AWborS/P8GwP+o90cCWGesW6+WxSCiC4hoKREtHRwcLKCLgiB0inqDw5TKZmwdGsavHt2Yc48C/vv3z4bvTQf/T759Pz7yo+Vqebr5//wtK/Hz5evDz3951WJcdtsqXPSzR3H94udx96rIvr396/fgop8/CgC49eEN+PwtK8LCcmPNqIw/ES0kohWOv/OMNpcAqAG4rpV9M/OVzDyfmefPmDFjNN0UBKHLaTBQzyif3PLQBlx4/XLsHak1bzxKGgy85phDw/cumnW7zhx7anhq81BsvTlNpHmMqKZQPrJSZTQbM/OZaeuJ6IMA3gXgbRzdHjcAOMpoNlstEwShR2kwo5FxJnQdgNWvecLMqJQCH9lX1bNZt5k51IZc59isbEReIYU8s33OAXARgD9i5r3GqgUAzieiASI6FsBcAEvy6ocgCN1Po8HIasu1Ucx6sxgNDQaU7U+ZxjG9Hw2O+rz+xX2J9XXPedSbDC4bLaPy/JvwTQADAG4nIgBYxMwfYeaVRHQjgMcRyEEXMnM9x34IgtDlMGc35trYZpWJRgMDKJcodlybtG6z5b3vryVNndf4621zqhuam/Fn5pelrLsUwKV5HVsQhPFFg9lrBJNt1WsBnj8zoxQ4r37NP8U4a6Mf9tlxA/HKPk3mERgtMsJXEISO00rAt1Gk58+m5+9v490+fPVLOK5kHmYOl+c1WYwYf0EQOk6DObOR055w1ieF0dBgRjn0/Fsf4WsHbbN6/macQDx/QRAmLMzZjbm+STTySX+3jgWomKVX3ElLxdSrbO3fxHXebMpgYvwFQZioNLj1bJ8iZJ8GM4gAInhd/LRe2N57VuNvymB55fmL8RcEoeO0kudfpOwDACUCCH75JYtxDjV/x63CtT0juh55naUYf0EQOk6jBdmnkbNHbB+LQCgR+bN6Mmj+UbZPso1b9omWi+cvCMKEpNW8fS7Q82c1yIuoPc8/0vz1a7Kt67zZkH3G3QhfQRCELLSat6/bFZXtQyAQUVupnlG2jz9zx1fyIZR9xPMXBGEi0qrnnzZgaqxhAFCav3+Eb4rnr1/ZXhLhzPMHwgC4aP6CIExIWvb8uTjPnxkokdb8PW3StleGPS1n35f7H3n+LXS4BcT4C4LQUVpN3YwklCKMP4OgNH/PzSZNlomyfHTbZBtnto8EfAVBmOhEAdxs7Rstth8NjCDVM9XzT9X8421chtz5BGMGfLN3tyXE+AuC0FFaLdFcpOwTDPIilefv0/z920cje/0STjPZJ68CdmL8BUHoKK3KPoUGfBnhCF9vtk+Kb25XaHCmerry/JH/+YnxFwSho7Qa8OUCPX9mGKmePs0/ZXvEdXtXU5d8ZZa7EM1fEIQJSeupnsUGfAPNf7SF3fxtfQFfyfYRBGFC02qtniJln0Yo+1Bbx7ONfjtVPcXzFwRhQtJ+wDe3LoUwWOX5+z1w0zjb0pAt9zizfZyF3STbRxCECU7ref7BazHZPqqcM8g/jSO73wPJQK9rF1LeQRCEnqT1PP8iNf9A8glmcmye6mm3sFM8M2f7sBR2EwRhgmMXP2vePngtJtvHHOHrb+N6H3wOXiNpK7l90zx/Mf6CIExEQmPejdk+MGv7uAOznOr5W6+OYzT1/HNS/cX4C4LQUVot0Vxknn84jSN8Rdnixtm+HyUnc3EFfJP7DWr7qG1yCmyL8RcEoaOE0kjmev7Ba5FVPX31/JnZ0vwt2cd65yzv4Bzha07jOE49fyL6NBExEU1Xn4mIvkFEq4noUSI6Ne8+CILQvbRe3qE42UcfIyjv4NLm07N9khO4Zwv4mhO4j8uALxEdBeDtAJ43Fp8LYK76uwDAt/PsgyAI3U1aMNTdPngtIs8fnF7Pn8EZR/j6Uz3d0ziO/3r+XwdwEeLnfB6AH3DAIgBTiWhWzv0QBKENtg0NY9f+aq7HaDXg22o5iNEQav7kL8PA1uf4+nTNv+SZJ8D0/MfdCF8iOg/ABmZ+xFp1JIB1xuf1apm9/QVEtJSIlg4ODubVTUHoKrbvGcHmXfs73Y2Q1/zbQrz+K7/L9RitBnBbHRE8GoJsH1XP3xOYjaV6ejR/Ntqb9JVLnptYVN4hr7OsjGZjIloI4HDHqksAfBaB5NMWzHwlgCsBYP78+fl/y4LQBZz65dsBAGu+8s4O9yRi9/5arvs3bXijwSgFI6qatu+Gev4NO9XTo/n7Zh/rL5e8nn+U55/PeY7K+DPzma7lRPRKAMcCeISCsdGzASwnotMBbABwlNF8tlomCEIPYhq3OjNKaGb8C8zzZ6Oev2u91Y9mef42fRW35z9uR/gy82PMfBgzz2HmOQiknVOZeROABQD+WmX9vA7ATmbemEc/BEHofmLGP4M3X2RtH+b0ev5Jz58T612vmr4y+ev5N9z7HCtG5fm3ya8BvAPAagB7AXyoA30QBKFLMG1bFm++1dTQ0RBU9YS3qidza56/S/N35vkb++1KzT8ryvvX7xnAhUUcVxCE7sc0iFm8+SIDvmE9f7jr+ftq+dif7Xx/jS/gG3j++Z6njPAVBKGjmEY1S65/kXn+QWE38s7ha3v+tpuus3+ibJ+k7OMz7o2cs33E+AuC0FHsgG8ziszz16mewUxe7vVx229r/kZDeGQfKyNItxuXAV9BEISsmEY1m+yjXgsK+CKlnn/Dru2TkH3igV775mDKPua5m7KPTOYiCD1EEZks3YJp3Lop4Kv7VQpH+LraWFU9rfVhTR/rsybI8w/em+djppCK7CMIPUS1kMI13UG3ev569wRV28cT8E1L9dSm2zeBe1+FjMCueWyZwF0QepKRnjL+rXn+RdXzj3n+8Hj+sLR6a31UzVN/Dt5U1CjmmOwT0/w5se1YI8ZfELqQaq1HjX+mbJ9iZJ/Q81cBX1fGTrPyDlF+f1zCKRvG3zWZTd16CsgDMf6C0IVUXdM7TVBief5ZNP9whqucPX9lqokoVs/fDvCmTeZi6/Z6H33lwPT2G56/eT41w/qL5i8IPcRIj3r+rQzyyj/gG7ySVdXTlqliBt/j+duaf+T5R5q/eT414zpIto8g9BC9pfmb77No/sFr3pdIH6dkVfWMlXNIeP72PuJGvxF6/knZJ+b5G/qXaP6C0EP0VrZPe55//tk+SvZB3PNPaPzWzcDETvFMeP4Vd8C3Zsh+eZ2mGH9B6EJ6SfbhNo1/7rKPei0FxX3cnj/SJ3D31fOvlCLN3zUhfV1kH0HoTXrK828xsyWUUIry/HVVz3C52cYa5OXN9om/VsqG5h8+yUTbmZq/eP6C0EP0lubfpZ5/GPAlECjhvQdtbM8/jl3WQb/G8vydAV/z+xfPXxB6hl5K9Ww14FvUNI7a2BOAUsnw3mNPKnYMIN4n+2lBv2rZR6d8NhocOx/R/AWhR+ldzb95+6KmcYyyfeL1/OPH5UQ1zvg+2HoNlmvZRz8B1Jlj+42neo76VJyI8ReELqQbNP+8Ao02rdb24YI8/0jzp9gcvvE8/3RRxtb6w4BvuYRyicLJ6uu2529V+MwDMf6C0IV0g/EvqrBoq7V9Qs0/7zx/9arr+UcG3GjD8cCz3X27qqemUiKUicKUT7OQGyAjfAWh56DAFnSF7JO3rOI6Tkt5/rnX9gkjviAkJRzdxhKBYvuwg8T6RlEpEUoloEyR5++7DpLqKQg9QJ8KBHZDtk9Rxr/l2j4FyT7ajpdSUj2DEb4ZPH/rCaBSDjx/Lfs0GvHzMQP+ovkLQg+gZYBuqOpZkO23qnpm0fyL8vyD12AOX3fAN1HVM7EXy/M3BnmVSgQV900EfOtGqqdo/oLQA+jsj25I9eyI59/CZC65p3oqwx16/lbQNmxn5f2bJDz/MNUz0PvLsYBvtJ14/oLQY5SVK9gdsk9Rx2k34FuQ508AQAlDrvsS1/zjREY/Hi8olSzZxwr42vP55oEYf0HoIrTn320B3zxLKcQ9/+btwyqYuef5R6meJXLLTc01f2X09XpETxJE5A34VhvjPNuHiD5ORE8S0UoiusxYfjERrSaiVUR0dp59EITxhP79d0OqpzmSNc9SCq1P4xi85j/CN3glQE3mEny2RyTHu2xJQkY7/UqkJR948/zrpuyX02lW8tktQERvBXAegFcx8zARHaaWzwNwPoATARwBYCERHc/M9bz6IgjjBW0kOm38mRl7Rmrh5zy9bNuYDtfqKBOhUnb7plFtn9y6BCBe26dEFMYA4lU9k9k/8X1wbDmz9vyV7KM8f/sJoohBXrkZfwAfBfAVZh4GAGbeopafB+AnavlzRLQawOkAHsixL4IwLtC/+Syyz96RGi67bRX+99kvx+SBsf0pP/DMNvz59xZH/crxXmTnt3/g6iU46YhD8Ll3zfO0131irN4yhP/76ydw6jGH4pktQ6gz47L3noyBShkA8PNl61FvMB5atwOXvPMVmJLhOjEzvnLbkzjtmGkAtERjBm+j/n7qhocxXI1LND9ctBZ3PLEZ5550OL511zMAgC27h/Gha5Zg0bPbQaBwdK++v3194VP4w1fNCvdTG+eyz/EA3khEi4nobiI6TS0/EsA6o916tSwGEV1AREuJaOng4GCO3RRshoZr+NIvVmJ/VR7GiqaVipWPrNuJa+9fg4fX7RjzfmzYsS/2OU/Zx67n/8KO/di4c3+i3d6RGvYM12IB3wee3YY7ntyC//jNKtz00Abc+vALWP9i1PcfLFqLi29+DD9e8jweXZ/tOu0ZqeO7dz+LO57cDMCYwD2UbqK2a7ftxUBfCbMPPUCdC3DdorW4a9UgPnvzCqzdtjdse+eqQeyr1kEEvPOVs/AXrz0Grz7qUADAwic2x4q5FRHwHZW7QEQLARzuWHWJ2vc0AK8DcBqAG4nouKz7ZuYrAVwJAPPnz+983lsP8eCa7bjmvjV45ytnYf6caZ3uTk8R6dnN2+aZ9WIbnDz1dVv2sfVvzbwv/AZAMAGKbusKRJvLzGqZWZ9edHttjEvhCN+ojyafOvN49JVLuPD65bERvr5rRgS8/cTIbH7wD+bgpuXrY+mdRaR6jsr4M/OZvnVE9FEAN3Fwu1xCRA0A0wFsAHCU0XS2WiZ0CXqAUe4jKIUErUxR6KoDP1bYN588s33iso+6AaSck3nTc/2P1tltgLNeJ32u5raB5q/Xx9uXShSW5WBubqy1zm9+bnA8zjPeyzvcAuCtAEBExwPoB7AVwAIA5xPRABEdC2AugCU59kNoEe115D1ZhpCkFdmn3sKNot1+2MfKg1hVTzXSNe2czGvkkkR8kknW66TPtdowPH/PNI5AUJ9Hm3M7cOuCrM/lUtBnc2xHtYDCbnkGfK8GcDURrQAwAuAD6ilgJRHdCOBxADUAF0qmT3eh//HyDPIJbsxgZtO2Dg917PoR32e+ef62TJNuQM1r5Dp38//WN3Aqjei6Bjsi8k/gDgBmUhIj/akFcHj+pWAqR5/nn9e1z834M/MIgL/0rLsUwKV5HVsYHdoDEc+/eLgVzz/HwU62oSwqz18PdsqSxln3yEMx2cfzvtl+gegJWGv+Ps9fyzZAssSzE8v1LxOh0eBYPaeY5p+p160jI3yFBJHnL8a/aFqpW5NnXfuE8S844Jvtycf9P+rzmjPLPtYTVTDIi7wB37Kh+QPNbzK2518OPX/zJmgWdsvU7ZYR4y8kkIBv52ilVr02+nl45UnZZ8wP4TyWNvxZ/vfsYmjO/bXh+etz1U5QOJOXI9UTUAFf9T7w/NP3b9n+UFIyNf/4NI7jL+ArjFMk4NsZ2CgVkMn45RjwtY1qvnn+8eP45Bwbr+wT8/bdy5vt12xPVj1/2xiXKSj5DCjNv8lxXJ4/gNi4mppU9RQ6wYjIPh0hZgRbKHA27vP8HXn5WQPezfL82xks5c7zN+v5x9uXLc+/2Y0rme0TLBmumZ6/me0jnr9QELq0gHj+xdJqgbN88/wt2aeoVM8Mef5h21YDvhmlK32u2gATgFLJP8hLp4ICqtZPkxsXOfL8Acvzj920svW7VcT4Cwm01imaf7HEjWCnZZ8iA75xY53V8/e1G6uArzbAwcyaUUZPWsCXM9y4bM1fp4qK7CN0nDDbRzz/Qml5UpMcPf8iZZ9wghNSUo5DOnHn83vy/Eeb6mnJPoSgnr9W/V15/lr4YU9fTUqOgC8A7K+6A74ymYtQGGHAVwZ5FUpc8+8uzz9v2adEgQetJzGx//fsEtd9ZUoJ+Jrv3fp/en8s2ceq6umSfRB6/s2/j7SA70ClpPqa/49PjL+QQAK+ncFOeWzaPseAbxbPe6xoMKOkattXa+4bmj2tZaVUaj3Pv1XPP8z2UfX8MwR8kWGEry/gu7/WQL8y/jXPOYwlYvyFBFUJ+HaE9gO+OfSlcM8/qHFf9Ywur1rzG1RSPH+v7NOq5x9m+0CN8E3uP1hvpHpyljx/f8BXz0NQkxG+QieQgG9naD3gq7YrIs8/RxWCwSAK8uW11NLM8+8rl7yDvEad568Hz4UjfClWzz+R52+meqJ1pylM9YzJPqL5Cx0g0vzF+BdJbFKTDJd+4gR8A++3VCKM1NznpOUgTSWc+zZp/X2ef6uyj3aC9ExeYaqndciyVdK51WtVNgK+2vjHqnpKto9QFCPi+XeEWI2bFgK+uWj+Rco+DY4Cvp7/PZfnD8QLoGl8Qd5W8/zDbXVVT2u9Rg8Cc61zYT85lELNvx5q/uO9nr8wTpFUz87Qruafi+xTaMBXef7kLypoZ/tUyoHBtG8KgP+6tO/521U94+31k4G5bRrJgHHwOuzz/DP1unXE+AsJRPPvDG1n++Qh+9iDvHIN+HJYM98X8LUntNeyT81h/H0T4rRd2weBd653l6b51zIcwy7XoAO+w17PP1O3W0aMv5Cg6tFdhXwxL3cmzz/HPP+iJ3MplXS2jza88TbJPP802Ue/tmf89bmGM3mVmnn+UZ5/lvx8V6ooEGj+2vjrY5dLJAFfoTgkz78zdJPnn8z2KUL2Ia/kmMjzV7KPfVMAjJuifQNrUfZJ1PP37Cfw/PWTSBbNP/45DPjW6uHk9PrYZeO4Y40YfyFBJPt0uCM9RizgmyXbh+OvY9uXAgO+3Dzga3v4laDgTkIOAvyD31rN8zdLOsfr+TuMfwuavy0bmWMETNlH71c8f6EwfLqrkC++UsQ+ii3sNuaHCGlwYADLJcJI3X1O9iCvvjTPPwz4Wssze/7xz6SC0ZHmH1+vA8JAVs0/jpZ9AISDvAA9T4BjgzEizwnchXFK1fMDFPJj0bPbQh0b6MLyDjk6Aqw8/xL5R5e7yjsAbs3fNxta1lOwj02AVc/f5fnrcQdZsn3s7aP32vMHgmqiwfzA+Vx7Mf5CAqnnXzx/c+2DOPMVMwEEXm2n6/kn0iQLqO2TLvtk1/xHm+1jn6tOQ400/3j70EPPeAzXk4PGNP5lQw7KA5F9hAQygXuxMDP2jtSxZ7gGIPBqu0/2KTbgaxs8X6qnO89f7bfNc7DbURDx9U/mUoqKtbXn+RvGv2x6/qQ8/0zdbhkx/kICyfMvFq0Ta0NWyRJOphsAACAASURBVOj5F1regYPa+Wd89S7ctmLjmB+LwoCv+5wSAd9ycjCUuT/XPjJn+zhG8Gr7HMyz7A/4ZtH8bQ2/bHn++lhlVVFOpnEUCkMmcC8WnR6o53DVRcuaEQ1Gyn6s3fur+OKClbFZo5z7dsg+W3bvx7ODe/C5W1ZmP2AGdG2fIOCbTfYJA7615HWyUzXt5c1wPUlF5Rs8ef7w1xpK7N9T3gEIzkvHD8oqkDzuZB8iOoWIFhHRw0S0lIhOV8uJiL5BRKuJ6FEiOjWvPgjt0St5/tV6AwseeSG32imZ+6EMhpY2KiXKZvw98kYaS9e+iGvvX4PHNuxM3zfbnxlbdg0DAKZP6c98vCzoVE9T9gHi55WUffyevy/bZ6w8/7RUz9Fm+/SVI8+/VCI1snj8ef6XAfgSM58C4AvqMwCcC2Cu+rsAwLdz7IMAYMvu/fjNyk1N2+0dqWHTzv1tp3puGxpOLHtq827s2l9taT9Fce/TW/GPP34Ij2/clcu+zdID+6t1r7etPf8Rw/PPlOffhuyjs2lcRtO1b/Pzxp37AQDTpwxkPl4WGobnb6Z0/n71VvyfXzyOoeGad5CXS/Mf64CvzvMHgB37qvjyLx+PrS8bqZ7taP5mwLevXAqfMvR+x6PmzwAOVu8PAfCCen8egB9wwCIAU4loVo796Gqe3LQL96/emusxfrp0PT7yo2XOATEm37n7Wbz7ivvCx8xW5IRla7fjtEsXYt32vbHl7/vOA7jm3jXe7Z7ctAs/XvJ8zLu5fvHzWLN1T6Jttd7AP//0kcQx2mXvSGCMtVFm5qbXyMe2oWG851v34frFz+OZwSH85VWLcc/TgwCAR9btwAmfvw3vvuI+57b6JmFq/i4jcuH1y3HZbU+Gn+2A79BwDe//7gP4h+uWefupPdOblm/A9+9fgwuvW+68abskk827AuP/EuX576/W8cUFK7Fzb7ab+xV3rsaV9zwTW7b42W34xSMvoMGMMlFM2//Ydctx9X3P4cu/eDwp+xiev+k5l42npnbHKtjbmZO1zP+3hbG5dgFV/kGt19fX7JNNYoSvFfAlQ/MPqomOP8//kwD+g4jWAfh/AC5Wy48EsM5ot14ti0FEFyi5aOng4GCO3ewc2/eM4Ft3PoPP3boi1+Psr9bB3Nzb2zY0jE3qBw6kywlPb96Nwd2R0di8axgNBrbtGQmXMTN27a+mev5X3PkMLr7pMVxx52oAwQ/vszc/hpse2pBou277Xvxs2Xo88Ow27/5+vmw9bly6zrluf7WOv/jeIqx8IZA89MQh2uB8/tYVOP5z/9OW3PX0liEsf34HPnvzY9g2FFyDoeHgprJq824AwJObdju31XVc9PdTKZHTm39y4y48tTnah+353/rwBix5bjt+/Zj/KU8f42fL1uNfF6zErx7biOsWP59oZx+/Znj+B0/qAwA8sXEXrr1/DRY/5/8+TBY+sRl3PLEltkzLTx/8gzkoleKe/G6V/fTCzn3Y8OK+2HYDfVGev9b/AWX8R1vewWpGBJxxwmGxZa8+emq8jd5WfSdmn2x85R2A4MZPoewTnzt4rBmV8SeihUS0wvF3HoCPAvgUMx8F4FMArmpl38x8JTPPZ+b5M2bMGE03u5IXduzDaZcuxO+fHsRwtT1vMyv6B9XM+Nt1SdLkhAt+uAyX3/FU+LnqOEa9wU1vOgf2BSMaH3p+R2x7lweujXTa/m54cB1ueNBt/LfsGsZ9q7fh4XU7Yseo1htgZvxoUWAEXVJCM8w+6ZTNrBJLzTrnvnLJeQOq1jkcAQskq08O7Q+O+5LJfk3eNSjKJeMkSyozNu0MDHAtvFlx7HMzqvVG4lrobf/0tKOd3vL0Kf2o1ht4YlNcmjNnvNKD48olQplo9OUdEp4/8IpZB+Prf/qqcNlVHzgt1ibU/Ova+PtNa6Kqp9E00PwN2cdIMR1rRjXIi5nP9K0joh8A+IT6+FMA31PvNwA4ymg6Wy3rKQZ3D6PeYLy4t5r6jzIW6IyIZkbN/mGmecC79lWxWxkbwDCkhtHOYqzDYKclfbi2CW8wKdLMSL3h9fBGrO3N/j29ZSh2nEl95eQOUogZ/5Fa5v4CRqqnalf2eP7VeiO2rzCwyc2vs6ufmoMmJc1AMtWyET4V1qzvKMtxgeD/0C5TVg1veBTTvjUH9lewv9rAU5uHMOuQSeHTh/n9hMZfxQ28VT3bDPhqv/7EIw4Jl9j3KbKyfdJ+076qnoCSffQxSjrbZ/zJPi8AeLN6fwaAp9X7BQD+WmX9vA7ATmYe28ThcYD5g8n642kXLW80qzho3xzSfiwjlhenDam5j9CgO9LxEtvVbMOcvCbRjSFtfw2vbl+1tjdln82G3JW2fx/mOe5Vco99PB92wLdSLoE5+aO3PWe7AFlYFyfle3PVv6+5pkJ0pFoOqScae1xC1jiJy/Ov1hthjr/b+JexatNujNQaMeMbN/7BdkE5BH95h6xyXlLzD16Pmz4ZAPDKIw9JTMJuZ/ukyz5WtpAZ8K1QzPMvdavn34S/A3A5EVUA7EeQ2QMAvwbwDgCrAewF8KEc+9C1jDgMZ15k9dDs9WmPyYGR5djn4DW5zGVcwjaWsU97WtBGspq2v3rD2+9QUrKMVnAulGjXCi7Pf8RxQ9TVGl3bDqvXvlKUU27akJFa3HhGgc34ftJu8iOOdc5ceUdtHX29fE9QzRipNyJx3OhPX7kUFnazmTxQwT4VkD92+oHh8gGjDELS8x9b2Ucb+kq5hNs++UbMOvgAOO5TsWOkyz5xSlaqp75GeVf1zM34M/O9AF7jWM4ALszruOMF8wfTjsbcCtpINzf+2QNk1To7n15Mz7KWRfbxGH3X00Iko6T3q5nxj25KHPazWjKeWNrI+DHPUWcR2fKIfl8uxSUlW/bRaYz2jSKh+avdhrKPNsYpN0eX5+9qby+q1jh6Yspwo3ahvXx7mS5p4PP8o/eRuTI9/3B7NSHM6AO+bs8fAE44PEhg1E9BGtvz73cY/75ykM2UFvA1NX/Snn+mXreOjPDtEHFtvJHrQKM0g+pqp/EZ0XojMLDNZJ8sx7U1/jTNP0vg2vaO4+vixqpqeLLDtWS/WyFu/C3Nv4nEZxtk7TXaxqqZ7JNFZnIFZ10xCdex9c3JvqlnN/6cuHFX641QInE5y6bxNw2+U/PXtXCsJyJN256//biCxAOMofn7PX9f+qc/1VP1Zxxq/kIKppFkzreOTvuyT/b9uQxPFmNtG/2apcfH2mbInnHpyr4+mucRl6va0PyNbfYozX8kPE76vpMTlSTLAzMzatYN1w746uPpm7Ozn4ah/5NTZ3v75CqvYAfl7ddmVB035sD4R8bbZrLh7R/QF5mrScb7vkqkkafKPhm/1kRJZ4fNtp9SIs+/EeuTSV/JbW5t2Sem+Zcg9fwnGskfAaPSWoJJC8fyG1RXO02zrJkRh7fsviGkGWu3F+mWfZJPF66+NdX8Q3kkkn0Io/T8jWthe/6u62Rify+6aJlphCLJJVrm8/z1e1teso81eSBY76yMaRx7oFKKBfgTnn+TJ0qNU/OvRamaLtlnwPDwDzCfAipJz19XwfTKPpkDvvHPLuNvL7NLOjs9f08QOC77kJXtI3P4TjhsXTlP3X+sZZ9ILkn38qsZYg22AcmU6pm2vxTZx97efJIwtxl1nr/S/F1PKq54gh2g1TKIaazsJyQgWcAsSwaZeYPXxtTV1jz2QKWEWj0a/RxmSWUcx2D2yeX56xr2Ls/fDOw2lX2U5z/aPH9bgnXdlBLG35rD12X8K17PP3rfVynFCruZ8wiMNeL5dwjby84z3TOr7GMbJp/H4Qr0hca7ljSiaTLKiJ3tkyXVs1nA1/vE4nnKqDdiQbhmefnu40bb7BuxUz3TjbK9TBuJusP4p8k+rhhM2rEGyiVUSuTsU8zz7yvHDHfVkrPSAsxmXxucvNHFNH+HkTVTJuPGP7KY/eYgrxKF8k67s5E56/lb2HEA2/N3BXwrPs3fLOlsaP66bEReirAY/w7h8oDyPlYrmn9/xV9WOKvmn032iXu0aRkkzfbHzLGSwLYnOWKlJpqyD3Nzw5mGuU04wtfYv8YVcLWXhdk+hrGy0ysBR56/I/U22c9oeV+5hL5yyXm+pj0fqJQwXGskvpssN2P7uLUGo9HgUOeu1jm82ZUcxtH0oA/wev5Wnn9Y1bNN2SeD558c5BWQludf8ck+nqqeeoaw8TjIS0jB9rKz6qZtHStjSp5pBCalGH+dGWMaomggmbEsU6pn/IkhMiz+rBSfLFNzeMrxY9lGKzpmM12+GfFsn7jsExv45ixbEV+mA4OmAba9bcCcxjG5H1+6qnkj6quU0FcmZ9u6JfvoOIa5j1ayfWKOQiN+rfsqkWxjYxp/l9Rjvm8e8M32G0tm+yTxDfJK0/x9nn+inr+u6qlH+GbqdeuI8e8QtgHLU/O387J9mD/QSX3lFNnHofk79P0sqYcuCQZwSy/N8sqzyiu2tGSPVh4epeyTKO/QpF+2FOLy/M0bifYE9WbaWGXR/M3/s8jzbyL7VMphBhMQGe/WjH8yeK237Q8996Rx7I8N5jILuMFYPsYB30S2T3PPX98iainlHbypnok8f3UMXdVTPP+JRXfKPobn31fOkDVTTy5zBIHTBk3p7RqM2NiBVM3fp2fX3AYm3N6OL5iDvDIYzjTM40XlHZLeuqtfdrZPmOfveZKpWbKGK+DrK7Zm3mj6y+Q1/rGAb18pvKGZx8mSfeXqvz3GJUr1TG5nesum8SxZGTJ6WWqef2bNP/7Zne3TzPN3yD6+gK9dzz8M+ELJPpm63TJi/DtEscY/q+xjev4lb150Vn0/0nn9x7XlljTj3+wm5hpgFt/eClSOoexjHjsq75DN88+S5+86N1+eP+C/4SY0/woljg8kUz33DKfJPs2tk+/66vIOQHPZxzSStk6utzdH+CYnc2naTQDZZB8b3SZV9vFo/vGqnvFzlFTPCUiR2T5RTZYWZR/vQKE0iadV2ScKzI7UG0Z6qEPzb/IkkVVbj1IWo2ONOG5krVCtRWUK9lnlHUZqUTqjy0u2R/i68vxrjqeHRJ5/Lcqc8UtjhuavZB9nnr/x3fdXyuHTTL8RILZltDTiBj/+FGDKNjZ9huzj9fwrhuxjaP55Bnxt7MlczH5rKq5HG9iF3UrhzaAUBnwzdbtlxPh3iESef44B36w/0pjxr/hln3A0rsrcCJY5bgi1yCilHXOyzjevJUeRxts20fwdaab2scx15g0rbqBb/y6q9UaYN2/Xu7fP0caWaNLy/M33UcA3Oo6uf5Ml1bOvUgqMuau8gxXwHVJPMwcOlI08/2xPlHZ/Epq/MULXRscDdOVPjV0SIVgWSCX2TVHTdsA3g+tve/4tpXomSjobAd8cUz3F+HcI2zjl6fmHBcxS5Bdmjv0oB/pK/oCv+QhvDfgxb2LNhv83GkHJAtNgZZF9fHp2M3klUUfIGuQ1ZaCSOL+sVOscGvjk8aJzzJLq2efw/F2yj/5+dLPYTcZzzc3/gTTN35Z99MfJ/RVnwLwZvu8mrvn7ZR9CmuwTL++gT3GsCru5Ar42WUo6+4w/UTR7VzLgC+SV7yPGv0PYBiZXzT9D6V17XZaAr7ldmOppGBdXVcvYflTbAwcig5X2lNKK7ONKnbU9VbOq50itERYRa1fzP3AgPmzGvJFNHvDvOyn7JDX/mLdc07IPYu2qdQ774DPI5nUJZB9y3pDief7xEgvV8Gkv/fs18UlyVUPzt2UfIsP4WyWf7SCp3n5syju04/nHn9Za0fyBaDSv9vb1sq6dxlFonyIDvlmybkyjTaQGeTWp7WPu0yn71CMj5bqR6PW6eFeQcpmi+TeRGZqVybZjEHaqZ+j5t6P51xuxQUj2/rXnnzY9pUZ7iLE8f4ekZeezjzSRlwDEcuxDzd+V529892a65eT+smMmr+bWyc7wMc8lqu0T36ZMFPP8yxkDvqPN9rFvGtk0/+C1lmb8Pdk+QHDjim50wbJwAncJ+E4s7B9MOzpz9mM199Bsj9CcCzW5P0eWT4rs4zu23sb0uM0JQ1wzWdnHd633HS/ct/VarQexBq3Zt6v59ysNPVxm3Kyic3TJPpbnX0rKPmmaf6MFzT82yKtcQn/FI/uYAV/DYz2wv2Kk5eonp+Y3S/Ppwh6M1+8p7xAYxEjzN21nrCSCEfBNy/bJUIUi2K6NbB97235XwNcj+wDB+fQbNzpABXwhnv+Ewx5I1I7OnAVdUwVIN/6moe4vl7zzyAJW/Z7ETFx+bddGL5s8EHnFabnqzWQGn3dpL/PJPv1lFQBty/Nn9JUppvVWw1HPZlzDJft4Ar4ZNX8z2ydNXrKX91e05p8uj5he7GSHRJflZmln+GhqKbJP3PO3ZB/Dcmmjqqc99JV3yFrYzb507Xn+2cs7AIGXH2YtadmnFMhdebmFYvw7RFGyj0uGadauv1JSP6Is+8xq/JPH1gbB9PzTbhijzfN3Zfno5SPKc+8rU9uF3YK8+VJsmX5NM8oJ2cc5yCv5FJD0/KObjFfzN5Zrzd/uky17mOdkBq7tG38avkyskbq/vEPZkEJAftlHT3dYtj3/drN9MtTzt9E6fdoE7qmyDyH2lANE5yWyzwSjM8Y/RfaJGQVCuZRS0tlh1J2VPpvkzic0/1o84yg565PStn0BX8cTiavfdm38mrrp9KfIIM3Q2/c5ZB8zmOzM80/IPsmAb81xXU3PXxe10zcZ3zy+5vWtlNx5/raRdHn++prpc2+GK9UzyDBrhHMW255/iRCmgZK13s78CSY+UQFf1Z1iA74BqZp/M8/fmtdAj1iWPP8Jhv2DyUvzb2aAXev6msk+jhtKlNZpSBUZjbGZ7WN7ha72fs3flEn8AWY7BbWqZJ9o0FMbmr+alCSm+Rv7Dz3mlH5pXKmeaZp/XaXMAkiVl4Lju2Qfy/O3NjU1/wP6HMH5Fgd5mf1njk/DaFK2gqC+Ug9lNbirTIFU4q3n367nn0H1z1LS2VfbR6+zbxi6sJuM8J1g2Aam87KPGeArxbTTRFvDI9exC21UYlU9G8kfvIm+IUw2DFaabt9MZoida6rn3wi9zvC4dVayT/uev66SGfbX2P+kviB/u5VUT9MIu0YgR7KPcSNtkq5q7ieUfWrpRrLi9Pyj69ey5m/dxDPJPoh7/mbbElGY6ZNe3qE9zz/FZockJnNxTOOYFvAtURQvEs9/gpMs6ZyP8XdNruKiNc8/WdAtbYIXwDNxuMPzz6L5myOL4/3KdrPRcyabNWq059+u7DOi5AvTWNXqHA5kSyulkKjn3yTbZ8SR56+vdTPNv+ZI9bTP1zZ+5jmZTzCt5PnH+x9/8vIFfEt2wNc0+Jbnr9Mi0wK+7cs+rXv+LtknLXBsP+XoZSDx/CccRWn+NY9unNafvorWTpvLK7bs45OZ0vLbozx/9m6f2Ldrgvcmxt9+CorJPmbAt13P39b8642wn6Ghdc5NHD+es7yD44nILO+QCJ57yoWY+/HdkGwjaco+YeC60aLmbw3sMreLUj3j25StVM+0ksh6gFQ84Btv136ef/NtbM3fFdxNu4eYN7oo2yffaRzF+HeI4jT/bLLPiJ3nX/LLPnaFRlNC8T1pZEn1NGv72H3Kci7xjJJ0bd2s4a+fOHS5g3bqLFXrjL4KxTJjag0OZbF+T2YN4Krn75/G0XwfjmQ1SmFP6gu+O680Fsvfdz/ppAV89UC2QKJLPu35SKsA69P8bYPo85x1fn9JBX7tmkeatvP8W4j41huN8EaUaNLE8++3PP8gz79LB3kR0fuIaCURNYhovrXuYiJaTUSriOhsY/k5atlqIvrMaI4/nimqvEPWUsWtyD62BKEDd/a6rKmekzOmevpyxV3HaOb5D1frRp85LOzWtuavZSPLfdUVPvtS6ugk6vnrbJ9Ynn/y3BqGkTMNadrTi/2EF7RNl0fi2T6R7GNKfs0MlFPzr8UlkkSef8ka4estjBbP+PHm+Weu7ZOpWYxQ82+wegJxtfET5Pnrp5woz79U6t6qnisAvAfAPeZCIpoH4HwAJwI4B8C3iKhMRGUAVwA4F8A8AH+m2vYcnQj4ZplUBTADvs3b6mCpe10TY6xH+BplFbJo/r79Nbs5xGvux+MWWrZpd5DXSD2Z6glE8/n2Vfyaf7Y8/+RTjTZmzNF326xMs2lIdNt6g2PebsLzN55m9MTpdmZWK7PE2dlW6QFfM7Dr3rd+KtA3gNEGfLPGBkxMzd/3lJKm+ZfMOkZqWZnyrec/qgncmfkJwPk4cx6AnzDzMIDniGg1gNPVutXM/Kza7ieq7eOj6YePHXtH8L7vPJDHrkfN9j3Dsc+3PLQBDzyzbcyPs68aGDki4JH1O3DW1+52thtSRoooyvMfqTec7Tft2h/WGf+3Xz2Ory98Ktx2x75quM2GHfvCdhf97NFQj7aPqT3/y+9YjT3DtXCbT/zk4dg2G3dEx/3TKxclsie27xkJ+3HN/Wuw4JEXYuvXbd8bbv+Bq5eEbXftr4Yph30VwrK1L3qvk4/hWgOVcjxoxwx88JoHAQTz8vZXSrh95ebEvte/uC+YeFz9xnW2z5d+8Ti+dntwbbfsHg73efnCp/GD+9eAGeGctR/+/tLgOOVAPrhp+Qbc+/TW2HFsI1IxPOuz//OehG6tMZ9mdPuP//ghDBnf1bmX35Nq3LbtGQnbXvX753Dz8g2hJObL8588UA6XTR6ohHbGPky5RKjogG+JsHVoBGd97W5sHRoOr49+PfNrdzdN3Fy7fW9sLuAs6H3uHanjwP6yU/MfcJR80OgxF+b5lfQgr8y9aI1RGf8UjgSwyPi8Xi0DgHXW8te6dkBEFwC4AACOPvrotjpRKhHmzpzS1rZ5c/zhB+Hckw7H6i1DOGhSH5at3Z7bsV533Evw2mOn4c5VW1LbveWAPpw2ZxqmTe7H9CkDWLttr9PrmDtzCl533EuwessQtg4FN7GTjjgEZ5xwGG5/YnMoAcydOQVvPn4Glq/dgd3DVecx33pgP06ePRV//6bjsO7FvUE/Xn4YHnxue2zqQAA4fuZBOHPeYbh71aA3m+Vlhx2EEgFPbd7t7PeZr5iJ3z+9FcO1Ok45amrYZwJwzkmH4xWzDsIhB/SlXicXLz/8IJx70iycPHsqpkyq4KxXzMTvntyCWqOBU4+eitfPnY5qo4H7Vm9NbGtep6kH9uGkIw7Bn7/2aOzYOxJrc8pRUzG4exgbduwLj3nWvJm4/fHNaDBj/pxDMX/ONHz0LS/F8udfdPZz3hGH4OwTZ2LTzv0gIrx93kw8uWl3ODJV86rZh+Btr5iJoeEa5s+ZhveceiReOmMKTp49Fe99zWzsHanhVUdNjZ1nGnMBnHD4wajVG1g9OBQunz/nUJx27DQAwOtfNh3vPuUInDx7Knbtr+J984/CwZP6cNE5L8c5Jx4OAPj8u+bh9S97CQDgM+eegKc27cbpx07DJ886HsdOnwxmYNe+KhiMuTOnYN6sg9FXLuHIQw/AbSs2ZfKi586cgjNOmIkHntmGo6cd6G3373/ySrzssMC+TJvcjw+/4Vhs3LkPJx5xCE49JvifPnn2VDAYj63fiY+/bS7mTJ+Mk448JLGvf3jrSzFtcj8A4IN/MAdHTt2Ms088HJUShQ7cWEPNtDoiWgjgcMeqS5j5VtXmLgD/zMxL1edvAljEzD9Sn68C8D9qu3OY+W/V8r8C8Fpm/lhaH+bPn89Lly7NfFKCIAgCQETLmHm+a11Tz5+Zz2zjmBsAHGV8nq2WIWW5IAiCUBB5pXouAHA+EQ0Q0bEInvqWAHgQwFwiOpaI+hEEhRfk1AdBEATBw6g0fyL6YwD/BWAGgF8R0cPMfDYzrySiGxEEcmsALmTmutrmYwB+A6AM4GpmXjmqMxAEQRBapqnm3w2I5i8IgtA6aZq/jPAVBEHoQcT4C4Ig9CBi/AVBEHoQMf6CIAg9yLgI+BLRIIC1o9jFdADJoZWdR/rVGtKv1ujWfgHd27eJ1q9jmHmGa8W4MP6jhYiW+iLenUT61RrSr9bo1n4B3du3XuqXyD6CIAg9iBh/QRCEHqRXjP+Vne6AB+lXa0i/WqNb+wV0b996pl89ofkLgiAIcXrF8xcEQRAMxPgLgiD0IBPa+HfTZPFEtIaIHiOih4lIT3ozjYhuJ6Kn1euhBfXlaiLaQkQrjGXOvlDAN9Q1fJSITi24X18kog3quj1MRO8w1l2s+rWKiM7OsV9HEdGdRPQ4Ea0kok+o5R29Zin96ug1I6JJRLSEiB5R/fqSWn4sES1Wx79BlXWHKv1+g1q+mIjmFNyva4noOeN6naKWF/a/r45XJqKHiOiX6nO+14uZJ+QfgpLRzwA4DkA/gEcAzOtgf9YAmG4tuwzAZ9T7zwD494L68iYApwJY0awvAN6BYBY2AvA6AIsL7tcXEcwSZ7edp77TAQDHqu+6nFO/ZgE4Vb0/CMBT6vgdvWYp/eroNVPnPUW97wOwWF2HGwGcr5Z/B8BH1ft/APAd9f58ADfkdL18/boWwHsd7Qv731fH+ycA1wP4pfqc6/WayJ7/6VCTxTPzCAA9WXw3cR6A76v33wfw7iIOysz3ALAnDfb15TwAP+CARQCmEtGsAvvl4zwAP2HmYWZ+DsBqBN95Hv3ayMzL1fvdAJ5AMCd1R69ZSr98FHLN1HnriXr71B8DOAPAz9Ry+3rp6/gzAG8jSpkNfuz75aOw/30img3gnQC+pz4Tcr5eE9n4H4nkZPFpP4y8YQC/JaJlFExODwAzmXmjer8JwMzOdC21L91wHT+mHruvNqSxjvRLPWK/GoHX2DXXzOoX0OFrpiSMhwFsAXA7gqeMHcxccxw77JdavxPAS4roFzPr63Wpul5fJ6IBu1+OPo81/wngIgAN9fklyPl6TWTj3228gZlPBXAuHw8zqgAAAk9JREFUgAuJ6E3mSg6e4boi77ab+gLg2wBeCuAUABsBfLVTHSGiKQB+DuCTzLzLXNfJa+boV8evGTPXmfkUBPN0nw7ghKL74MLuFxGdBOBiBP07DcA0AP9SZJ+I6F0AtjDzsiKPO5GNf9ok8oXDzBvU6xYANyP4QWzWj5HqdUun+pfSl45eR2berH6wDQD/jUimKLRfRNSHwMBex8w3qcUdv2aufnXLNVN92QHgTgD/C4FsoqeONY8d9kutPwTAtoL6dY6Sz5iZhwFcg+Kv1+sB/BERrUEgT58B4HLkfL0msvHvmsniiWgyER2k3wN4O4AVqj8fUM0+AODWTvRP4evLAgB/rTIfXgdgpyF15I6lsf4xguum+3W+ynw4FsBcAEty6gMBuArAE8z8NWNVR6+Zr1+dvmZENIOIpqr3BwA4C0E84k4A71XN7Oulr+N7AfxOPUkV0a8njRs4IdDVzeuV+/fIzBcz82xmnoPATv2Omf8CeV+vsYxWd9sfgmj9Uwj0xks62I/jEGRZPAJgpe4LAp3uDgBPA1gIYFpB/fkxAjmgikBL/LCvLwgyHa5Q1/AxAPML7tcP1XEfVf/0s4z2l6h+rQJwbo79egMCSedRAA+rv3d0+pql9Kuj1wzAyQAeUsdfAeALxu9gCYJA808BDKjlk9Tn1Wr9cQX363fqeq0A8CNEGUGF/e8bfXwLomyfXK+XlHcQBEHoQSay7CMIgiB4EOMvCILQg4jxFwRB6EHE+AuCIPQgYvwFQRB6EDH+giAIPYgYf0EQhB7k/wP73GvEBqD32wAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u2HaGRVEYGQS",
        "colab_type": "text"
      },
      "source": [
        "## 測試"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5yFuUKKRYH73",
        "colab_type": "code",
        "outputId": "b18389e2-83e2-430a-90cb-8c34f3f93014",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "agent.network.eval()  # 測試前先將 network 切換為 evaluation 模式\n",
        "\n",
        "state = env.reset()\n",
        "\n",
        "img = plt.imshow(env.render(mode='rgb_array'))\n",
        "\n",
        "total_reward = 0\n",
        "\n",
        "done = False\n",
        "while not done:\n",
        "    action, _ = agent.sample(state)\n",
        "    state, reward, done, _ = env.step(action)\n",
        "\n",
        "    total_reward += reward\n",
        "\n",
        "    img.set_data(env.render(mode='rgb_array'))\n",
        "    display.display(plt.gcf())\n",
        "    display.clear_output(wait=True)"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aex7mcKr0J01",
        "colab_type": "code",
        "outputId": "0ffbf013-226f-40f0-faae-735f48493ed3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(total_reward)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wUBtYXG2eaqf",
        "colab_type": "text"
      },
      "source": [
        "## 參考資料\n",
        "\n",
        "以下是一些有用的參考資料。\n",
        "建議同學們實做前，可以先參考第一則連結的上課影片。\n",
        "在影片的最後有提到兩個有用的 Tips，這對於本次作業的實做非常有幫助。\n",
        "\n",
        "- [DRL Lecture 1: Policy Gradient (Review)](https://youtu.be/z95ZYgPgXOY)\n",
        "- [ML Lecture 23-3: Reinforcement Learning (including Q-learning) start at 30:00](https://youtu.be/2-JNBzCq77c?t=1800)\n",
        "- [Lecture 7: Policy Gradient, David Silver](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ]
}